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SUMMARY 


A research experiment was performed to assess situation awareness and workload of 
commercial airline pilots while monitoring simulated autoland operations with three advanced 
display concepts: two enhanced electronic flight information system (EFIS)-type display 
concepts and one totally synthetic, integrated pictorial display concept. All display concepts were 
presented on a heads-down, projection-based display system. In addition, the EFIS-type display 
concepts also incorporated a simulated monochrome head-up display (HUD) presented in an 
over-the-glareshield fashion. The basic instrument arrangement for the EFIS-type displays was a 
conventional T-arrangement of instruments with a Boeing 757 duplicate of a PFD over a ND. 
The three advanced display concepts incorporated visual enhancements consisting of a sensor- 
derived wireframe runway and iconic depictions of sensor-detected obstacles (other traffic) in 
different locations on the display media. The first enhanced EFIS-type display concept 
incorporated the sensor-based wireframe runway and icon obstacles into it’s monochrome HUD; 
while, the second enhanced EFIS-type display concept incorporated the sensor-based information 
into it’s primary flight display (PFD). The advanced pictorial display concept incorporated 
pathway-in-the-sky guidance and sensor-based wireframe runway and icon obstacles head-down 
in an integrated, large field of view (62-degree horizontal) display. Each of the enhanced visual 
and advanced pictorial display concepts was simulated under Instrument Meteorological 
Conditions (IMC). A conventional EFIS-based display concept utilizing a HUD without any 
sensor-based infomiation provided an experimental control condition for the simulation. This 
conventional EFIS-based/HUD display concept was simulated under Visual Meteorological 
Conditions instead of IMC because the pilot had to visually detect the runway and other traffic 
without the aid of a sensor. Comparisons were made between the conventional EFIS-based/HUD 
concept and the advanced display concepts to ensure that the advanced concepts did not degrade 
a pilot’s current level of SA or increase his or her workload during autoland operations. Using a 
fixed-base simulator, eight commercial airline pilots repeatedly flew complex microwave landing 
system (MLS)-type approaches to closely spaced parallel runways with extremely short final 
segments. Various scenarios, involving conflicting traffic situation assessments, main display 
failures, and navigation/autopilot system errors, were used to assess the pilots' situation 
awareness and workload during autoland approaches with the four display concepts. Situation 
awareness was operationally defined as the pilot’s knowledge of ownship position relative to its 
desired flight route, the runway and other traffic as well as the pilot’s comprehension of ownship 
systems information. From the results, for each scenario, the integrated pictorial display concept 
provided the pilots with statistically equivalent or substantially improved situational awareness 
over the other display concepts. Similarly, subjective results indicated a very strong preference 
for the integrated pictorial display concept over the two enhanced EFIS-type display concepts 
and the conventional EFIS-type display concept for all monitoring tasks (i.e., the approach, 
verifying the runway location, detecting ground runway incursions, monitoring autopilot 
functions, monitoring airborne traffic, etc.). All pilots indicated that the advanced pictorial, 
“pathway-in-the-sky” display concept afforded substantial improvements in situation awareness 
for detection and comprehension of an abnomial flight condition (e.g., conflicting traffic, loss of 
localizer signal, etc.) over the two enhanced EFIS-type display concepts, hi addition to increased 
situation awareness, subjective rankings indicated that the pictorial concept offered reductions in 
overall pilot workload (in both mean ranking and spread) over the two enhanced EFIS-type 
display concepts. Out of the four display concepts flown, the pilots ranked the pictorial concept 
as the display that was easiest to use to maintain situational awareness, to monitor an autoland 
approach, to interpret information from the mnway and obstacle detecting sensor systems, and to 
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make the decision to go around. 


INTRODUCTION 


Many research programs have been established by the government and industry to study 
aircraft cockpit technologies that may enable improved crew simation awareness, safety, 
operational efficiency and reduced crew workload during critical mission phases. (See ref. 1.) 
Several of these programs involve the use of "synthetic vision" to enhance transport operations 
under restricted-visibility conditions. Synthetic vision also provides the cornerstone technology 
for more advanced aircraft, such as a high-speed civil transport, that, because of the complex 
aerodynamic and economic requirements, may have limited forward visibility. 

Numerous studies have been undertaken to assess the requirements (ref. 2) and to determine 
the performance (ref. 3) of various synthetic vision system concepts aimed at providing or 
enhancing some of the capabilities mentioned above. One such area of investigation involves the 
use of synthetic vision concepts to enhance crew situation awareness (SA) as a means of 
providing increased safety and advanced capabilities. 

Initial investigations have been conducted on flight deck displays aimed at studying the 
spatial awareness component of situation awareness (refs. 4-6) and its effect on flight path 
control during the landing approach portion of the flight mission. (See ref. 7.) Results of these 
studies indicate that intelligently designed, enhanced flight display formats are able to increase 
situation awareness (at least some of its components) without incrementing pilot workload or 
degrading flight path control. A follow-on experiment was designed to assess the SA provided 
by enhanced display formats where the subjects task was focused on approach monitoring rather 
than flight path control. 

This paper will focus on the aforementioned experiment whose objective was to assess pilots' 
situation awareness and workload while monitoring autoland operations, under Instrument 
Meteorological Conditions (IMC), utilizing three advanced display concepts: two enhanced 
electronic flight infomiation system (EFIS)-type display concepts and one totally synthetic, 
integrated pictorial display concept. The hypothesis to be tested was that large-screen, pictorial 
displays offer enhanced performance and safety through increased situational awareness (without 
incrementing pilot workload) over conventional and advanced EFIS-type displays. Situation 
awareness was operationally defined as the pilot’s knowledge of ownship position relative to its 
desired flight route, the runway and other traffic as well as the pilot’s comprehension of ownship 
systems infomiation. All display concepts were presented on a heads-down, projection-based 
display system. The EFIS-type display concepts also incorporated a simulated monochrome 
head-up display (HUD) presented in an over-the-glareshield fashion. The basic instrument 
arrangement for the EFIS-type displays was a conventional T-arrangement of instruments with a 
Boeing 757 duplicate of a PFD over a ND. The three advanced display concepts incorporated 
visual enhancements consisting of a sensor-derived wireframe runway and iconic depictions of 
sensor-detected obstacles (other traffic) in different locations on the display media. The first 
enhanced EFIS-type display concept incorporated the sensor-based wireframe mnway and icon 
obstacles into it’s monochrome HUD; while, the second enhanced EFIS-type display concept 
incorporated the sensor-based information into it’s primary flight display (PFD). The advanced 
pictorial display concept incorporated pathway-in-the-sky guidance and sensor-based wireframe 
mnway and icon obstacles head-down in an integrated, large field of view (62-degree horizontal) 
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display. Further explanation of the display formats will follow in the Experiment Design section 
under the subheading Display Concepts. After being compared to each other, the three advanced 
display concepts were then compared to a conventional EFIS-based display concept utilizing a 
F1UD without any sensor-derived information. The EFIS-based/HUD concept represented a 
conventional flight deck in today’s fleet. Comparisons were made between the conventional 
EFIS-based/FlUD concept and the advanced display concepts to ensure that the advanced 
concepts did not degrade a pilot’s current level of SA or increase his or her workload during 
autoland operations. This conventional EFIS-based/FlUD display concept was simulated under 
YMC instead of IMC because the pilot had to visually detect the runway and other traffic without 
the aid of a sensor. Various scenarios, involving conflicting traffic situation assessments (both on 
the ground and in the air), main display failures, and navigation/autopilot system errors, were 
used to assess the pilots' situation awareness and workload during autoland approaches with the 
four display concepts. The intent of these abnormal scenarios was to determine if significant 
differences exist between the display concepts that incorporate sensor-based wire-frame runway 
and icon obstacles (advanced display concepts) when measuring a pilot’s situational awareness 
and workload under IMC, and then compare these display differences to a conventional EFIS- 
based/HUD display with no sensor-based information under VMC. 


Situation Awareness Assessment Tools and Techniques 

Finding techniques to assess situation awareness is a challenging task within itself. Several 
techniques have been suggested in the overall literature, each with its own advantages and 
drawbacks. Historically, the most common method used to assess situation awareness is by 
measuring traditional pilot/vehicle performance. However, there has been no established direct 
relationship between performance and awareness and, therefore, these measures should be 
supplemented by additional techniques. (See ref. 8.) Three additional techniques (ref. 9) that 
were used to assess situation awareness in this experiment were Anomalous Cues/Detection Time 
Techniques, Freezing/Probes and Subjective Methods. Brief descriptions of these techniques are 
presented below. 


Anomalous Cues/Detection Time Technique 

This technique requires setting up scenarios that introduce slowly developing problems that 
may require some subject interaction. The experimenter then measures the time it takes for the 
subject to detect the problem, as well as the time before any corrective action is taken. 


Freezing/Probes 

This method entails a direct approach in which the experimenter either interrupts a task or 
“freezes” the task and then proceeds to take some form of measurement. Usually, the 
experimenter asks the subject relevant questions (in effect, probing them) concerning the task the 
subject was performing. (See refs. 10-11.) Often questions are asked as to future events (based 
on what has transpired until the moment of “task freezing”) that may provide greater insight as to 
the subject’s awareness of the situation at that moment, hi other words, the better the SA, the 
more accurately the subject will be able to predict the immediate future, hi addition, after 
resuming the task, other measurements indicative of SA may be taken (such as time to restore to 
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some predetermined condition). These methods require caution in that not only has the original 
task been corrupted, but also the probe results must rely on the subject’s short-term memory. 


Subjective Methods 

Subjective methods mainly consist of questionnaire type evaluations where the subject, either 
verbally or by handwritten means, expresses personal opinions or feelings about the topic of 
situation awareness. 


Mental Workload Assessment Measures 

For the purposes of this paper, mental workload is defined as the amount of processing 
capability required by a person to perform a specific task. The supposition is that there are 
performance decrements when a task requires greater processing capability than is available 
within an individual. In other words, when mental workload is exceeded, performance declines. 
Two subjective measures, modified Cooper-Harper (C-H) ratings and questionnaire display 
rankings, were utilized in this experiment to assess the pilot’s mental workload while monitoring 
autoland operations with each of the display concepts. 


Modified Cooper-Harper Ratings 

The modified C-H scale (fig. 1 and ref. 12) provides a subjective assessment of mental 
workload associated with perceptual, cognitive, and communications tasks. The 
modified C-H scale is typically administered to a subject after an experimental run or 
group of experimental tuns is completed. 


Subjective Methods 

Subjective methods mainly consist of questionnaire type evaluations where the subject 
rates the level of mental effort required to accomplish a specific task. 
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Difficulty level Operator demand level Rating 



Very easy, 

Operator mental effort is minimal 

highly desirable 

and desired performance is easily 


attainable 


Easy, Operator mental effoit is low 



Figure 1 . Modified Cooper- Harper scale for mental workload assessment. 
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ABBREVIATIONS 


ADI 

attitude direction indicator 

AGL 

above ground level 

ANOVA 

analysis of variance 

AP 

autopilot 

C-H 

Cooper-Harper 

CH 

Conventional HUD 

CRT 

cathode ray tube 

DERP 

design-eye reference point 

EC 

Embedded Conventional 

EFIS 

electronic flight information system 

EH 

Enhanced HUD 

FD 

flight director 

FMS 

flight management system 

FOV 

field of view 

GPS 

global positioning system 

GS 

glideslope 

HUD 

head-up display 

ILS 

instrument landing system 

IMC 

Instrument Meteorological Conditions 

INS 

inertial navigation system 

LNAV 

lateral navigation 

LOC 

localizer 

MLS 

microwave landing system 

ND 

navigation display 

Nmi 

nautical miles 

OTW 

out-the-window 

PF 

pilot flying 

PFD 

primary flight display 

PNF 

pilot not flying 

PV 

Pictorial Vision 

RA 

resolution advisory 

SA 

situation awareness 

SAL 

standard approach to landing 

SNK 

Student-Newman-Keuls 

TA 

traffic advisory 

TCAS 

Traffic Alert and Collision Avoidance System 

TOGA 

take- off/go- around 

VISTAS 

Visual Imaging Simulator For Transport Aircraft Systems 

VMC 

Visual Meteorological Conditions 

VNAV 

vertical navigation 
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METHODS 


Subjects 

Eight current line pilots with national commercial airlines, all with extensive glass-cockpit 
experience and familiarity with the Traffic Alert and Collision Avoidance System (TCAS), acted 
as subjects in the experiment. (See table 1.) Subjects were asked to complete a brief 
questionnaire describing their flight experience. The number of years flying commercial aircraft 
that subjects reported ranged from five to 17.5, with a mean of 11.6 years. Six of the eight 
subjects also had experience flying military aircraft, with a mean of 10.3 years. The total number 
of hours flying ranged from a low of 5,300 to a high of 11,000, with a mean of 8,338 hours 
flying. A summary of the flight experience of the pilots serving as subjects is given in table 1 . 

Table 1 . Summary of pilot experience of subjects in Situation Awareness experiment. 


Subject 

Years Flying 
Commercial 

Years Flying 
Military 

Total Hours 
Flying 

767 F/O 

6 

20 

8000 

767 F/O 

14 

— 

8700 

777 F/O 

12 

3 

7500+ 

757 Captain 

17.5 

5 

10250 

737-300 Captain 

17 

— 

11000 

DC-9 F/O 

5 

13 

5300 

767 F/O 

7 

10 

5500+ 

757/767 Captain 

14 

11 

10450 





Means 

11.6 Years 

10.3 Years 

8338 Hours 


Simulator Description 

The Langley Research Center has developed a flexible, large-screen flight display research 
system, named the Visual Imaging Simulator for Transport Aircraft Systems (VISTAS), which 
was utilized for this experiment. The workstation-based simulator is comprised of the following 
elements: visual system hardware, graphics generation hardware and software, aircraft 
mathematical model, and computer implementation. (See fig. 2.) The visual and interactive 
control elements of this flight display research tool have been integrated as a piloted workstation 
in order to explore the advantages and limitations of large-screen, pictorial, reconfigurable 
display concepts and associated interactive techniques. 
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Out-the-window view 
with simulated Head- 
up Display (HUD ) 


Head-down 

Reconfigurable 

Display 


Figure 2. Visual Imaging Simulator for Transport Aircraft Systems. 


Simulator Visual System 

The core of the visual system is embodied in dual, full-color, high-resolution cathode ray tube 
(CRT) projectors that are configured to vary the projected display’s aspect ratio by edge- 
matching and overlapping the images from each projector. Since each of the two projected 
images is 15 inches in height by 20 inches in width (standard 3:4 aspect ratio), a maximum 15 by 
40-inch image can be achieved. Image resolutions up to 1280 x 1024 pixels in a 60-Hertz 
progressive scan format can be obtained (per projector). The images are generated by dual 
graphics display generators operating in conjunction, utilizing the same visual database in order 
to produce a single, large-screen, integrated picture (combined by the projection system onto the 
rear-projection screen that serves as the simulated aircraft’s main instrument panel). Given that 
the design-eye reference point (DERP) for transport cockpit applications is typically around 28 
inches, the full 40-inch wide display provides a maximum 71 -degree horizontal field of view 
(FOV). The DERP is centered about the full 15-inch x 40-inch display. At the stated maximum 
resolution this yields approximately 1 minute 40 seconds of arc of horizontal angular visual 
resolution (about 36 pixels per degree) and approximately 1 minute 45 seconds of arc of vertical 
angular visual resolution (about 34 pixels per degree). 

In addition, an out-the-window (OTW) representation of the real-world forward view (forward 
cockpit window) is simulated by a high-resolution projector and front projection screen 
combination. A 40-degree horizontal FOV is achieved by placing the 80-inch wide, curved 
projection screen 9.2 feet from the pilot eye reference point. The 1280 by 1024 pixel resolution 
projected image yields a 1 minute 52 second of arc of horizontal angular visual resolution (about 
32 pixels per degree). HUD symbology is presented overlaying the OTW view and sized so that 
its subtended angular presentation corresponds to what it would be if it were an actual glareshield 
HUD. 


Simulator Audio System 

To add realism to the simulation, audio feedback and verbal cueing were provided by a desk- 
side computer with a sound system that allowed playback of digitized sounds (files) timed to 
external events. Audio feedback was provided in the fomi of aircraft sounds such as background 
engine noise and tire "chirps" on touchdown. A series of verbal announcements was utilized to 
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represent various pilot flying (PF), pilot not flying (PNF) and tower communications on final 
approach such as landing clearances, instruments crosschecks, wind conditions, critical altitudes 
and flap settings, hi addition to adding realism, the intent was to also assist in maintaining 
vigilance during the exhaustive set of repetitive approach trials. 


Aircraft Mathematical Model and Computer Implementation 

A simplified six-degree-of-freedom mathematical model of a transport aircraft with a 20-Hertz 
update rate was used in this study to provide the interaction between the pilot and the flight 
display formats. The linear transfer functions and gains were obtained empirically to represent a 
fixed-wing generic transport aircraft with rate command controls. Turbulence was used in the 
simulation to make it harder for the pilot to detect when a system error had been induced versus 
nomial deviations from path due to wind gusts, as occurs in actual flight. The turbulence was 
introduced into the mathematical model through the addition of gust components to the body- 
axis longitudinal and lateral velocity variables. The level of turbulence was considered to be 
moderate by the participating pilots. 


Cockpit Layout 

The pilot workstation was configured as the pilot side of a generic transport, fixed-wing 
aircraft with the pilot’s seat designed to position the subjects so that their eyes were at the DERP. 
The workstation also accommodated the dual-head projection system and the rear-projection 
screen that simulated the instrument panel. Pitch and roll inputs to the aircraft mathematical 
model were provided in the workstation by a two-degree-of-freedom sidearm hand-controller 
with spring centering. A throttle lever that utilized a voltage-referenced potentiometer as the 
signal source provided throttle inputs. Typical self-centering rudder pedals provided yaw inputs. 

The display screen (instrument panel) was tilted so that the center of the screen's display 
surface was set perpendicular to the pilot's line-of-sight. It's height was set so as to provide a 17- 
degree view over the top of the screen (line-of-sight from horizontal), which is typical of over- 
the-glareshield views in many aircraft. 


Monitoring Task 

The pilot’s task was to monitor, not control, autoland operations during a standard approach 
to landing (SAL) using the display concept available. The SAL was 6.0 nautical miles (Nmi) in 
length, consisting of a microwave landing system (MLS)-type approach (fig. 3) to closely-spaced 
parallel runways. The short final approach segment was only 1.7 Nmi in length. The SAL had a 
20-knot headwind on base and turbulence throughout the entire approach. The turbulence was 
implemented in the simulation so that a crosswind gust component was present on each leg of the 
approach. On the SAL, the ownship began at 150 knots, 2150 feet above ground level (AGL) 
altitude, and landed at 130 knots. Autothrottles and the autoflight system were engaged 
throughout the duration of the SAL. The simulation began with the ownship in vertical 
navigation (YNAV) and lateral navigation (LNAV) modes with glideslope (GS) and localizer 
(LOC) amied. Typically, a pilot would manually amt the GS and LOC modes but for the 
purposes of this simulation these modes were automatically armed. The GS and LOC modes 
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were captured about 50 seconds into the run. The total run time for the SAL was approximately 
2 minutes and 55 seconds. The SAL used a 3-degree glideslope on base and final with a 4.5- 
degree glideslope in the turn to final. Ownship flare initiation began at 75 feet AGL altitude and 
the SAL ended at a threshold crossover altitude of 50 feet AGL. Other traffic was always present 
on the SAL in the fomt of an aircraft landing on the parallel runway 18 seconds before the 
ownship, runway traffic holding short on the approach end of the active runway, and crossing 
traffic on the ownship ’s base leg (at 90-degree angles). The crossing traffic was initially ahead 
and at the ownship’ s same altitude but then initiated a climb at a point in the simulation to 
prevent it from causing a TCAS Traffic Advisory (TA). 

Eleven separate experimental scenarios, utilizing the same SAL procedure, were designed for 
the data collection sessions. The first of these eleven scenarios was a normal mn in which no 
anomalous situations occurred during the SAL. The remaining 10 scenarios employed the SA 
assessment techniques referred to in the Introduction to generate an anomalous situation during 
the SAL. These 10 scenarios include a Flight Director Conflict with Autopilot Scenario, Flight 
Director Conflict with Raw Data Scenario, Aircraft Incursion on Final Scenario, Flag Take- 
off/Go-around (TOGA) Scenario, two Navigation System Error Scenarios, three Blanking 
Scenarios and a Probe Approach Scenario. All of the awareness scenarios involving abnormal 
situations in the investigation were completed well before a 50-foot threshold crossover altitude 
was obtained. 


8L Curved Approach 8R Curved Approach 



iso kts 139 kts (Drawing Not to Scale) 

2150ft 1700ft 



1 39 kts 1 50 kts 

1700 ft 2150 ft 



Figure 3. SAF Approach Plate for Runway 8L and Runway 8R. 


Scenarios Based on Situation Awareness Assessment Techniques 


The techniques chosen for this experiment were based upon their success in measuring 
situational awareness in a previous study (ref. 7) and their ability to generate suitable scenarios in 
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the context of transport approach and landing operations. The pilot’s basic, or standard, task was 
monitoring (not controlling) an approach to landing pattern designed for this experiment. Being 
a monitoring task, the traditional lateral and vertical RMS errors, as well as control input data, 
were not recorded during the basic task. However, pilot button presses indicating 
abnormalities/uneasiness with the autoland operations were recorded. The pilot could push two 
buttons (located next to his right hand) throughout the duration of a mn. The first button was 
labeled ‘CONCERNED’ and was to be pressed if the pilot detected an abnormality while 
monitoring the autoland procedure. The second button was labeled ‘TOGA’ indicating that the 
pilot would disconnect the autopilot and fly manually; thus terminating the mn. If the TOGA 
button was not pressed, the mn ended upon crossing over the runway threshold. 

As mentioned previously, all of the abnormal scenarios were implemented within the standard 
approach to landing procedure. Six scenarios were developed in order to utilize the anomalous 
cues/detection time technique. In addition to the slowly developing system anomalies, these 
scenarios included crossing traffic situations that sometimes caused TCAS alerts. Two types of 
probe techniques were also employed. The first technique interrupted the SAL procedure by 
blanking the displays and then introduced a new task - flying with a backup instmment (only the 
ADI, or attitude direction indicator, portion from the EFIS PFD). The blanking scenarios were 
unique in this experiment because this is the only time the pilot manually controlled the aircraft 
instead of monitoring its autoland operations. The supposition to be tested was that a superior 
display format would allow the pilot to think ahead of the airplane and thus be able to continue 
flying based on retained information. This scenario was thus formulated as a “Think-Ahead” 
awareness tool. The other probe technique also used the SAL procedure but generated three 
abnormal events in succession on the display. After the third event occurred, the display blanked 
to black and relevant questions about spatial orientation, instrument readings and traffic 
awareness were then verbally presented. Finally, numerous subjective questionnaires were 
administered in which the subject evaluated the display concepts subjectively by answering 
relevant questions and by ranking the display concept based upon the perception of the situation 
awareness afforded. Unsolicited subject comments were also recorded throughout the trials. 


Anomalous Cues/Detection Time Scenarios 

In the experiment, six scenarios (described below) can be categorized under this SA 
assessment technique. For each display concept encountered by the pilot, a base leg TCAS TA 
was embedded within two of the anomalous cues/detection time scenarios. The scenarios utilized 
for the embedded TCAS TA were the Flag TOGA scenario, the Runway Incursion on Final 
scenario, the Autopilot Oscillation scenario and the Navigation System Error scenario. (See table 
3 in the Procedures section-Organization of Trials.) So, in addition to the slowly developing 
system anomalies, these four scenarios included crossing traffic situations that sometimes caused 
TCAS alerts. 


Flight Director Conflict with Autopilot Scenario 

The basic approach pattern (to the left parallel runway, 8L) always included another aircraft 
flying a SAL to the right runway. The other aircraft landing on Runway 8R landed 18 seconds 
ahead of the ownship. Crossing traffic in front of the ownship on base and runway traffic 
holding short of the active runway were also present for this scenario. The intent of this scenario 
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was to implement a slow bias into the simulation so that the autopilot was not correctly 
following the flight director (fig. 4). This bias was implemented by incrementing a sinusoidal 
offset with time to the roll command (fig. 5) which is output from the flight director and used as 
input to the autopilot. The bias was slowly diverging (37 seconds) with a limited amount of 
offset (188.5 feet off path). The bias started at 15 seconds into the run and the run ended either 
upon a TOGA press by the pilot or upon crossing the runway threshold. Each pilot was exposed 
to this scenario once per display concept. 



Figure 4. Guidance for FD conflict with AP Scenario. 



Figure 5. System Schematic of Bias Error in Flight Director Conflict with Autopilot. 

Flight Director Conflict with Raw Data Scenario 

The basic approach pattern (to the right parallel runway, 8R) always included another aircraft 
flying a SAF to the left runway and landing 18 seconds ahead of the ownship. Crossing traffic in 
front of the ownship on base and mnway traffic holding short of the active runway were also 
present for this scenario. The aim of this scenario was to implement a slow bias into the 
simulation so that the autopilot was correctly following the flight director but the flight director 
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was not correcting back to the appropriate path (raw data-derived). (See fig. 6.) The bias was 
implemented by incrementing a sinusoidal offset with time and adding this term to the lateral 
error input to the flight director (fig. 7). The bias was slowly diverging (35 seconds) with a 
limited amount of offset (477 feet lateral error). The bias started at 15 seconds into the run and 
the run ended either upon a TOGA press by the pilot or upon crossing the runway threshold. 
Each pilot experienced this scenario once per display concept. 



Figure 6. Guidance for Flight Director Conflict with Raw Data Scenario. 



Aircraft Incursion on Final Scenario 

The basic approach pattern (either to mnway 8R or 8F) always included another aircraft flying 
a SAF to the mnway parallel to the active mnway and landing 18 seconds ahead of the ownship. 
Crossing traffic was found on the base leg of the ownship ’s SAF. The purpose of this scenario 
was to have ground traffic, that normally holds short, pull onto the approach end of the active 
mnway during the ownship ’s landing. Each pilot saw two replicates of this scenario (1 to 
mnway 8R; 1 to mnway 8F) for each display concept. 
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Flag TOGA Scenario 


The basic approach pattern (either to runway 8R or 8L) always included another aircraft flying 
a SAL to the runway parallel to the active mnway and landing 18 seconds ahead of the ownship. 
Crossing traffic in front of the ownship on base and mnway traffic holding short of the active 
mnway were also present for this scenario. The intent of this scenario was to simulate a flight 
situation on final approach where the pilot would automatically be forced to disconnect the 
autopilot and fly a go-around approach. For this scenario a fault was generated by the loss of the 
localizer signal at 650 feet AGL altitude. The fault created a mandatory TOGA situation typically 
indicated by a warning flag on the PFD. When the localizer failed, the autopilot continued 
following the glideslope guidance (vertical path and descent rate). The autopilot followed the 
lateral inertial track since it no longer had a valid localizer signal. Indications on the display of a 
localizer failure included displaying PFD and navigation display (ND) warning flags, removal of 
localizer raw data, removal of flight director commanded vertical bars, and drawing amber lines 
on the PFD mode enunciator. For the advanced pictorial display concept, there was no vertical 
flight director bar so the Flight Director circle remained centered laterally. Each pilot 
experienced two replicates of this scenario (1 to mnway 8R; 1 to mnway 8L) per display concept. 


Autopilot Oscillation Scenario 

The basic approach pattern (either to mnway 8R or 8L) always included another aircraft flying 
a SAL to the mnway parallel to the active mnway and landing 18 seconds ahead of the ownship. 
Crossing traffic in front of the ownship on base and runway traffic holding short of the active 
mnway were also present for this scenario. The purpose of this scenario was to build three full 
glideslope oscillations starting at an altitude of 580 feet AGL and increasing linearly to a 
maximum deviation of 1 dot error at threshold crossover. Each pilot was exposed to this 
scenario two times (1 to mnway 8R; 1 to mnway 8L) per display concept. 


Navigation System Error Scenario 

The basic approach pattern (either to mnway 8R or 8L) always included another aircraft flying 
a SAL to the mnway parallel to the active mnway and landing 18 seconds ahead of the ownship. 
Crossing traffic in front of the ownship on base and mnway traffic holding short of the active 
mnway were also present for this scenario. The aim of this scenario was to generate a navigation 
system error between the autopilot and the onboard sensor on final approach. The onboard sensor 
was implemented to simulate a device that could detect mnway edges and obstacles (including 
traffic) regardless of weather conditions. To accomplish this position conflict, a lateral position 
error was introduced into the inertial navigation system/global positioning system (INS/GPS) 
position infomration. (See fig. 8.) For this scenario, there was a disagreement between where 
the autopilot “saw” the mnway and where the sensor “saw” the mnway (fig. 9). The 
disagreement of the two systems from mnway centerlines could be a small one (75 feet) or a 
large one (300 feet). Each pilot was exposed to one large disagreement and one small 
disagreement between the navigation system and the sensor system for each display concept. 
These two runs were flown to opposite mnways. 
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Figure 8. System Schematic of Bias Error in Navigation System. 
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Figure 9. Guidance for Navigation System Error Scenario. 

Freezing/Probes Scenarios 


In the experiment, four scenarios (described below) can be categorized under this SA 
assessment technique. 


Blanking Scenarios 

Three different blanking scenarios exposed each of the eight pilots to incidents of simulated 
display system failure (by blanking the display screens) for each display concept. The pilot’s sole 
source of information with which to begin flying the ownship in these blanking scenarios was a 
backup instmment (the ADI portion from the EFIS primary flight display). Prior to the 
experiment, a flight path visualization tool was developed to evaluate the amount of time 
subjects tended to stay within 1-dot glideslope and 1/2-dot localizer errors after display screen 
blanking. The average time was empirically determined to be approximately 20 seconds for the 
blanking scenarios’ setup and spatial configuration, i.e. banking, descending turns. Therefore, 
tracking data was collected for twenty seconds of backup instrument flying (data collection 
started at blanking: either 3 seconds before path turn initiation, 3 seconds before path turn rollout 
initiation or 2 seconds after a crossing traffic conflict is generated, as discussed in the following 
blanking scenario descriptions). 
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Blanking Scenario one emulated a display system failure just before the turn to final (three 
seconds before turn initiation). Similarly, Blanking Scenario two occurred just before rollout on 
final (about three seconds before rollout initiation). The simulated display system failure in 
Blanking Scenario three occurred two seconds after a TCAS Resolution Advisory (RA) appeared 
while on base leg of the SAL. The RA was generated by the crossing base leg traffic not 
climbing fast enough. The commanded TCAS maneuver generated by the RA was a descent 
between 500 to 1500 feet per minute which was displayed on the vertical speed indicator by 
turning it green in the allowable zone and red in the forbidden zone. Each pilot saw two 
replicates of each blanking scenario for each display concept. 


Probe Approach Scenario 

The probe technique of freezing the task during execution and conducting an extensive quiz to 
assess subject awareness was implemented within the SAL procedure. Prior to the screen freezing 
and subsequent blanking, three events were generated in succession on the display. First, a 
lateral bias was introduced into the system as the ownship approached the turn to final causing a 
flight director conflict with the autopilot (event 1). The lateral bias became large enough to 
generate a TCAS TA on final approach with another aircraft landing on the runway parallel to the 
active runway (event 2). Two seconds after the TA appeared the aircraft lost it’s glideslope 
signal (event 3) causing a TOGA warning flag to appear in the PFD and ND. Similar to the 
localizer failure scenario, the loss of the glideslope caused an amber line to be drawn through the 
PFD mode enunciator and removal of the affected raw data deviation bar and flight director 
command bar (horizontal). When the glideslope failed, the autopilot followed the vertical inertial 
track but continued to follow localizer guidance (lateral path). For the advanced pictorial display 
concept, the flight director is a circle, not command bars, so the flight director circle remained 
pinned vertically where the failure occurred. Four seconds after the glideslope failure (86 
seconds into the run), the screen blanked and the subject was given the probe questionnaire. 
Each pilot received only one probe scenario across the four display concepts encountered during 
the two days of testing. Therefore, for the entire research experiment, only 8 probe scenarios 
were administered (one per evaluation pilot). 
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Experiment Design 


Independent Variables 
Hypothesis 

Large-screen, pictorial displays offer enhanced performance and safety through increased 
simational awareness (without incrementing pilot workload) over conventional and advanced 
EFIS-type displays as measured by increases in detection times and decreases in reaction times to 
developing system anomalies and traffic conflict situations. 


Display Concepts 

This experiment attempted to assess pilots’ SA and workload while monitoring autoland 
operations utilizing three advanced display concepts incorporating sensor-based wire-frame 
mnway and icon obstacles, as well as a conventional EFIS-based display concept for baseline 
comparisons. 

The baseline EFIS display, which also incorporated a simulated monochrome (green) HUD 
presented on the simulated OTW view, used no sensor-based information and was simulated 
under VMC. (See fig. 10.) The EFIS+HUD baseline display concept will hereinafter be referred 
to as the Conventional HUD (CH) display concept. The basic instrument arrangement (for the 
CH concept and first two advanced display concepts) was a conventional T-arrangement of 
instmments with a B-757 duplicate of a PFD over a ND (See fig. 11.) To the left of the PFD was 
a typical airspeed indicator dial and to the right were typical altitude, vertical speed, and turn 
coordinator instmments arranged over one another in that order. Non-standard (but used also in 
the first two advanced display concepts) was a power indicator that integrated engine and 
ambient information to display actual power (including engine spool-up) in percent thrust. (See 
ref. 13.) Also presented on the power indicator were power commanded by the throttle setting 
and power desired by the Flight Management System (FMS) for flying the programmed 
approach. 
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Figure 10. Out-the-window HUD symbology used in Conventional HUD display concept. 



Figure 11. Basic EFIS instalment arrangement. 

The HUD symbology (fig. 10) included digital altitude, groundspeed, airspeed and vertical 
airspeed readings, pitch and roll scales (in degrees), wind direction and speed, and a horizontal 
heading tape. Both groundtrack and heading markers were present on the horizontal heading 
tape. The central HUD symbology consisted of a flight path symbol, flight director, 
localizer/glideslope raw error lines, and pitch attitude symbol. At 300 feet radio altitude, a 
symbolic mnway (dashed green outline) appeared on the HUD and was overlaid on the mnway 
as perceived on the simulated OTW forward view. The symbolic mnway location depiction can 
be thought of as being derived from ENS/GPS position information. The HUD was attitude- 
centered and had a 40° horizontal FOV. 

The first advanced display concept (hereinafter referred to as the Enhanced HUD (EH) display 
concept) incorporated a sensor-derived ainway depiction and icon obstacles (other traffic) into 
the Conventional HUD display concept and was simulated under IMC. (See fig. 12.) The sensor- 
derived ainway was depicted as a solid green outline on the monochrome HUD; while the HUD 
symbolic mnway was depicted as a dashed green outline. Also displayed on the HUD were the 
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sensor-based icon obstacles that used the same shapes and symbology as the TCAS objects 
shown on the ND. Similar to the OTW HUD, the sensor horizontal FOV was 40 degrees. As 
mentioned in the description of the CH display concept, the basic head-down instrument 
arrangement for the EH display concept was a conventional T-arrangement of in struments with a 
B-757 duplicate of a PFD over a ND. (See fig. 11.) 



Figure 12. Enhanced HUD display concept. 

The second advanced display concept (hereinafter referred to as the Embedded Conventional 
(EC) display concept) embedded the sensor-based runway and icon obstacles into the EFIS PFD 
instead of on the OTW HUD. Hence, the 40° FOV of the sensor was “minified” because it was 
represented in the smaller area of the PFD as opposed to the actual 40° FOV of the HUD. For 
the EC display concept, there was no out-the-window view because the HUD was removed, but 
the HUD symbology for the flight path symbol, flight director and pitch attitude symbol were 
embedded into the PFD. (See fig. 13.) Within the PFD, the sensor runway was displayed as a 
solid magenta outline and any sensor icon obstacles were displayed with their appropriate TCAS 
color and shape. Similar to the EH display concept, the symbolic mnway appeared as a dashed 
white outline in the PFD at 300 feet radio altitude. A perspective mnway, displayed as a solid 
green rectangle, was also seen in the compressed 40° FOV of the PFD. 
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Figure 13. Embedded Conventional display concept. 

The third advanced display concept (hereinafter referred to as the Pictorial Vision (PV) 
display concept) presented the pilot with a head-down, integrated pictorial display format that 
combined a computer-generated OTW view with overlaid HUD symbology. (See fig. 14.) The 
PV display concept had a 62° horizontal FOV as compared to the 40° FOV of the other three 
display concepts being examined. The OTW portion of the display consisted of a pathway-based 
approach tunnel, depicted by magenta dashed lines that represented the comers of the tunnel 
"box". The bottom of the tunnel was further marked by a magenta bar, spaced at approximately 
l/10th nautical mile intervals, that also incorporated a vertical post to provide a relative (above 
ground) altitude cue. The tunnel’s width and height corresponded to fractions of lateral and 
vertical instrument landing system (ILS) beam errors (1/2 dot right/left and 1 dot above/below 
tunnel centerline, respectively). The HUD symbology included airspeed and altitude vertical 
tapes, roll and pitch scales (in degrees), as well as a horizontal heading tape. All of the tapes 
incorporated Flight Management System (FMS) command “bugs.” A vertical speed indicator 
was integrated onto the altitude tape as a growing/shrinking barber pole with a digital vertical 
speed tag (whose position on the altitude scale would denote the altitude to be attained in one 
minute based on current vertical speed). Similar to the CH display concept, the central HUD 
symbology for the PV concept consisted of a flight path symbol, flight director, 
localizer/glideslope raw error lines, and pitch attitude symbol. The display was attitude-centered 
with rate command control, although the pilots attempted to monitor the flight path vector. A 
“smoked-glass” (see-through) navigation display (fig. 15) was presented on the left side of the 
pictorial display, basically duplicating the EFIS ND. Thus, the see-through ND provided 
horizontal situation display infomiation by depicting traffic within the OTW display FOV 
(delineated by the acute lines about the ownship centerline) as well as traffic outside the FOV. 
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Figure 14. Partial view of the Pictorial Vision display concept’s computer-generated out- 
the-window view overlaid with HUD symbology. 



Figure 15. See-through navigation display on Pictorial Vision display concept. 


In order to evaluate spatial awareness, scenarios were constmcted that required the use of 
TCAS Il-like implementations. All four display concepts incorporated TCAS symbology, but the 
implementation differed with respect to the display placement of the TCAS information. The 
CH, EH and EC displays incorporated TCAS symbols on the ND, along with relative altitude 
tags and vertical direction arrows (if climbing or descending). The meaning of the symbology 
itself is defined, for purposes of this experiment, in table 2. hr actual field service, the TCAS 
advisory algorithms have changed and their implementation have become more sophisticated 
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since the inception of this experiment. For all displays, the TCAS escape guidance command to 
either Climb or Descend was implemented on the Vertical Speed Indicator (VSI) in the form of a 
color-coded command bar. The pilot responded by keeping the VSI needle (or arrow) in the 
green-colored portion of the indicator (and out of the red). When this was achieved, the pilot was 
following the TCAS escape guidance command at an appropriate vertical rate. Warnings and 
commands were strictly visual (no audible alerts). The EH and EC displays also incorporated the 
TCAS symbology on the HUD thus providing a visual cue for relative traffic position OTW. For 
the PV display, TCAS symbology with the appropriate warning-color and shape-coding was 
implemented on the see-through ND and on the computer-generated traffic in the OTW scene. 
For all four display concepts, the TCAS advisories were turned off below 500 feet above ground 
level (AGL) altitude, although unfilled blue square symbols were used to represent other traffic 
on the displays, and their positional infomiation continued to update. 


Table 2. TCAS II Symbology 


Symbol 3 

Definition 

O Unfilled blue diamond 

Other Traffic: 

> 7000 foot relative altitude or 

> 7 Nmi range at closest point of 
approach (CPA)._ 
Nonthreatening 

♦ Solid blue diamond 

Proximity Traffic: 

Within 1200 foot relative altitude and 
< 6 Nmi range at CPA. 
Nonthreatening 

• Solid yellow circle 

Traffic Advisory: 

Within 1200 foot relative altitude, 
<2 Nmi range at CPA, and 
time to CPA • 45 seconds. 

■ Solid red square 

Resolution Advisory: 

Estimated miss distance • 750 ft, 
<.l Nmi range at CPA, and 
time to CPA • 30 seconds. 


“Hie actual display has these symbols in color; for puiposes of this report, the symbols are in black and 
white. 


Dependent Measures 

All of the scenarios involving abnomial situations were implemented within the standard 
approach to landing procedure. SAL’s with no abnomial situations developing (hereinafter 
referred to as Nomial Runs) were randomly distributed in the test mn sequence to lessen the 
pilot’s expectancy of abnormal events occurring within each mn. Therefore, no performance 
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metrics were utilized for the Nomral Runs. 


The intent of these abnormal scenarios was to determine if significant differences exist 
between the display concepts that incorporate sensor-based wire-frame runway and icon obstacles 
(Enhanced HUD. Embedded Conventional, Pictorial Vision) when measuring a pilot’s situational 
awareness under IMC, and then compare these display differences to a conventional flight 
display (Conventional HUD) with no sensor-based information under VMC. 


Metrics for the Anomalous Cues/Detection Time Scenarios 

In each of these scenarios, the objective performance metrics were the time to detect and time 
to react to an anomalous situation. Detection time was recorded when the pilot pressed the 
‘CONCERNED’ button and reaction time was recorded when the pilot pressed the ‘TOGA’ 
button. An additional metric for five of the six anomalous cue/detection time scenarios was the 
time difference between the CONCERNED and TOGA button presses. The one excluded 
scenario was the Flag TOGA scenario because it was assumed that detection and reaction 
occurred simultaneously since the pilot was required to immediately execute a TOGA. For those 
anomalous cue/detection time scenarios with an embedded TCAS TA on base leg, a second 
objective performance metric was the time to detect the Traffic Advisory indicated by a pilot 
‘CONCERNED’ button press. 


Metrics for the Freezing/Probe Scenarios 
Blanking Scenarios 

These are the only scenarios where the pilot was required to take control of the airplane and 
continue flying the approach. After blanking all displays (simulating display system failure), the 
pilot continued flying the approach solely utilizing the back-up instrument for approximately 20 
seconds for each scenario. At blanking, the turbulence was turned off but the wind remained on 
so that the airplane maintained it’s inertial track until the pilot provided manual control input. 
This was done in order to maintain initial conditions for each of the display conditions for this 
scenario. For each blanking scenario, the objective performance measures were: 

- vertical (altitude) path error mrs, mean, and standard deviation 

- lateral path error rms, mean, and standard deviation 

- distance from path rms, mean, and standard deviation 

Probe Scenario 

After being exposed to the Probe scenario, the pilot was immediately asked a series of 
questions about the pilot’s situation awareness the moment the screen blanked. No objective 
performance metrics were utilized for the Probe Scenario. 


Metrics for the Subjective Methods 

Six questionnaires (one for each display concept, one comparing the three display concepts 
flown in IMC, and one for the probe scenario) were administered to each pilot during this 
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simulation experiment. (See the right hand portion of table 3.) 

As shown in table 3, each pilot was asked to complete a questionnaire at the end of the data- 
gathering runs for each display concept which dealt with the general evaluation (e.g., advantages, 
disadvantages, ease of detecting abnormal fight situations, etc.) of that particular display concept. 
(See appendix C.) Immediately after the Probe scenario was encountered, the researcher asked 
each pilot a series of questions (appendix C) about the pilot’s spatial awareness the moment the 
screen blanked. The researcher recorded the pilot’s verbal comments on a questionnaire 
(appendix D). After completing all the runs for each display concept, the probe questionnaire, 
and the individual display concept questionnaires, the pilots completed a final questionnaire that 
involved detailed comparisons of the three display concepts (EH, EC, and PV) flown in IMC. 
(See appendices C and D.) 

Another method of subjective assessment of the individual displays was given in the form of 
modified Cooper-Harper (C-H) ratings for mental workload assessment. (See fig. 1 and ref. 14.) 
After the data-gathering mns for each display, each pilot issued a modified C-H rating for 
monitoring autoland approaches with that particular display concept. 


Procedure 

The experiment encompassed a total of two eight-hour days with scheduled rest periods. On 
the first day, after being briefed on the purpose of the experiment, the details of each Display 
Concept, and the various Scenarios to be encountered, the pilot spent about forty minutes going 
through familiarization training on VISTAS to learn about it’s handling characteristics of the 
airplane model. The Conventional HUD display concept was used for the simulator 
familiarization. The pilots were then thoroughly trained with the standard approach to landing 
procedure, and then were thoroughly exposed to each abnormal Scenario (except the Probe 
scenario), for each Display Concept. The second day was the data collection session. 


Organization of Trials 

The four Display Concepts (CH, EH, EC, and PV) were randomly blocked across pilots, and 
the experimental scenarios were randomized within each Display Concept block. (See table 3.) 
Due to its intrusive nature as a SA assessment technique, the Probe scenario was always the last 
run in the Display Concept block in which it appeared. Pilots flew all 1 1 scenarios (some with 
replicates) for each display concept. If a scenario was flown to both rnnway 8L and 8R, these 
mns were considered replicates since the paths were mirror reflections of each other. For 
example, in table 3, the symbols A, and A 4 are considered replicates of each other because each 
refers to the same experimental mn of an aircraft incursion on final approach, flown to parallel 
mnways. Varying the approach paths by using parallel rnnways helped minimize pilot 
complacency and expectancy. Within each Display Concept block, two TCAS TA’s were 
embedded within some of the anomalous cue/detection time scenarios. So, in addition to 
detecting the TA, the pilot was still required to detect and avoid an anomalous flight condition 
during the autoland approach. Therefore, these mns with an embedded TA were considered 
identical to the same mns without the embedded TA (same basic scenario) for the purposes of 
detecting and reacting to an anomalous condition. For example in table 3, the symbols A, 
(without embedded TA) and T 3 (with embedded TA) represent the same experimental run of the 
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aircraft incursion on final approach to Runway 8L. The TA was for detection purposes only and 
did not affect the rest of the run. Table 3 presents an outline of a typical session, the details of 
which varied from pilot to pilot. 


Table 3. Typical Pilot Session. 


Display 

Concept 

Approach 

Conditions" 

Questionnaires 

Individual 

Display 

Final 

Probe 

EH 

A 1 ,N 1 ,T 8 ,A 4 ,A 9b ,N 1 ,A 5 ,A 7 ,B 2 ,A 2 ,N 2 ,B I ,N,,A 6 ,T 3 ,B 3 ,A 10a 

✓ 



PV 

N 1 ,A 8 ,T 6 ,A 1 ,B 1 ,A 2 ,A 10a ,A 4 ,N 2 ,A 9b ,A 5 ,B 3 ,N 2 ,N 1 ,B 2 ,T 7 ,A 3 

✓ 



EC 

A 4 ,N 2 ,A 1 ,B 2 ,A 9a ,N 1 ,A 7 ,T 5 ,B 3 ,A 8 ,N 2 ,A 3 ,A„A 6 ,B 1 ,N 1 ,T 10b ,P 

✓ 


✓ 

CH 

B 2 ,T 9a ,A 8 ,A 3 ,A 5 ,N 1 ,A 1 ,T 4 ,A 6 ,N 1 ,B 3 ,A 7 ,A 10b ,N 2 ,B 1 ,A 2 ,N 2 

✓ 

✓ 



"Conditions: 

Nj , signifies a standard approach (Normal Run) to either Runway 8L or 8R 

A : signifies a flight director conflict with autopilot while on approach to Runway 8L 

A, signifies a flight director conflict with raw data while on approach to Runway 8R 

A 3 4 signifies a aircraft incursion on final approach to either Runway 8L or 8R 

A 5 6 signifies a Flag TOGA (localizer failure) while on approach to either Runway 8L or 8R 

A, 8 signifies an autopilot oscillation while on approach to either Runway 8L or 8R 

A 9a9b signifies a small navigation system error (75 feet) while on approach to either Runway 8L or 8R 

A 10a i 0b signifies a large navigation system error (300 feet) while on approach to either Runway 8L or 8R 

B 123 signifies the three blanking scenarios 

T 34 signifies an embedded TA in aircraft incursion on final scenario to either Runway 8L or 8R 
T 56 signifies an embedded TA in Flag TOGA while on approach to either Runway 8L or 8R 
T 78 signifies an embedded TA in autopilot oscillation while on approach to either Runway 8L or 8R 
T 9a9b signifies an embedded TA in small navigation system error while on approach to Runway 8L or 
8R 

T 10al0b signifies an embedded TA in large navigation system error while on approach to Runway 8L or 
8R 

P signifies probe on final approach 
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EXPERIMENTAL RESULTS AND DISCUSSION 


All of the scenarios under investigation (excluding the Probe scenario) were designed as full- 
factorial, within-subjects experiments, with Display Concept and Replicates (where applicable) 
as the factors. The data collected in the experiments were analyzed using repeated measures 
analyses of variance (ANOVA) for each metric. Since extensive pilot variability is expected, the 
repeated measures design was chosen because in this experimental design each individual acts as 
his or her own control eliminating any post-treatment effects attributed to individual 
characteristics. The particular repeated measures design used was a randomized block design, 
with each individual designated as a “block”. (See ref. 15.) Student-Newman-Keuls (SNK) tests 
(at a 5-percent significance level) of individual means were performed at various stages in the 
analyses. 


Objective Data Analyses 

Anomalous Cues/Detection Time Scenarios 

The intent of the anomalous cues/detection time scenarios was to determine if significant 
differences exist between the advanced display concepts that incorporate sensor-based wire-frame 
runway and icon obstacles (Enhanced HUD, Embedded Conventional, Pictorial Vision) when 
measuring a pilot’s situational awareness under IMC. These displays were then compared to a 
conventional EFIS display concept (Conventional HUD) with no sensor-based information under 
VMC. The meteorological conditions were established for the display concepts according to the 
scenarios as discussed in the Introduction. The scenarios developed for this simulation 
experiment required the pilot to detect anomalous situations with the ownship with respect to 
traffic and the runway position. Since the pilot had no sensor-derived information in the 
conventional EFIS display concept, he had to detect traffic and the runway visually. Hence, this 
concept was simulated under VMC instead of IMC. Table 4 lists the repeated measures ANOVA 
analyses perfomted for the anomalous cues/detection time scenarios. In each of these scenarios, 
the performance metrics were the times to detect and react to an anomalous situation by pressing 
the ‘CONCERNED’ button and the ‘TOGA’ button, respectively. Pilot detection time (in the 
form a CONCERNED button press) of a TCAS Traffic Advisory was also a performance metric 
in four of these scenarios. Additional analyses were perfomted on the time difference between 
the CONCERNED and TOGA presses for five of the anomalous cues/detection time scenarios. 
The one excluded scenario was the Flag TOGA scenario because it was assumed that detection 
and reaction occurred simultaneously since the pilot was required to immediately execute a 
TOGA. Only those scenarios with significant statistical differences between factors will be 
discussed in the remainder of this section. A complete summary of the ANOVA analyses 
perfomted for the anomalous cues/detection time scenarios are listed in appendix A. 
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Table 4. List of ANOVA analyses performed. 


Anomalous 
Cue/Detection 
Time Scenario 

CONCERNED 

press 

(detection time) 

TOGA 

press 

(reaction time) 

Time difference 
between 

CONCERNED and 
TOGA presses 

TCAS Traffic 
Advisory Press 
(detection time) 

Flight Director 
Conflict with 
Autopilot 

✓ 

✓ 

✓ 


Flight Director 
Conflict with 
Raw Data 

✓ 

✓ 

✓ 


Aircraft 
Incursion on 
Final 

✓ 

✓ 

✓ 

✓ 

Flag TOGA 


✓ 


✓ 

Autopilot 

Oscillation 

✓ 

✓ 

✓ 

✓ 

Navigation 
System Error 
(small) 

✓ 

✓ 

✓ 

✓ 

Navigation 
System Error 
(large) 

✓ 

✓ 

✓ 

✓ 


Flag TOGA scenario 

Significance of the sphericity test (p < 0.05) on one of the main factors, displays, indicated 
that the repeated measures ANOVA results should not be accepted. Sphericity is a form of 
deviation from the assumption of homogeneity of variance to which repeated measures designs 
are especially sensitive. (See ref. 15.) Investigation of the objective data showed that 6 out of 7 
outliers for the TOGA button press reaction time could be attributed to one pilot. This one 
subject’s reaction times were increasing the likelihood of unequal variances. A repeated 
measures ANOVA (appendix A. 14), excluding this particular pilot, showed marginally 
significant differences among the main factor displays (F(3,18) = 2.77, p < .072), but not in the 
other main factor, replications, or in the interaction between these two factors. Post hoc analysis 
using SNK test with significance level .05 showed no significant differences between the three 
sensor-based display concepts. However, the Embedded Conventional display concept did show 
differences when compared to the perfomiance obtained with the Conventional HLTD display 
concept (no sensor infomiation present). Figure 16 graphically presents the results of the Flag 
TOGA scenario. The mean TOGA press time (or pilot reaction time to an anomalous situation) 
for the EC display was 1.9 seconds sooner (corresponding to 23 ft relative altimde difference) 
than the CH concept. 
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Figure 16. Flag TOGA scenario mean TOGA press times per display concept. 


For this scenario, the indication of a localizer failure appeared both in the PFD and the ND for 
the CH, EH, and EC display concepts. The inference from these results is that the pilot’s 
reaction time was quicker with the EC display concept probably because his visual scan 
(attention) was not divided between the head-up and head-down displays or across a wider field 
of view like it was with the other display concepts. The pilots may also have been visually 
distracted by the VMC scene found only in the CH display concept, resulting in display 
differences between that and the IMC EC display concept. Although marginally significant, the 
display differences in the Flag TOGA scenario would not be critical in real-world flight 
operations, having a relative altitude difference of only 23 feet. 


Autopilot Oscillation scenario 

Repeated measures ANOVA analyses (appendix A.15-A.16) on CONCERN and TOGA 
button press detection and reaction times with displays and replications as independent variables 
showed a highly significant main effect for displays (F(3,21) = 7.95, p < .001 and F(3,21) = 
17.80, p < .000, respectively), but no significant main effect for replications or interaction effects 
between the two main factors present. Post hoc comparisons of the means (using the SNK 
technique) for both the CONCERN and TOGA button press times showed no differences 
between the EH, PV or CH display concepts but there was a difference between these concepts 
and the EC display concept. Figures 17 and 18 graphically present the results for the displays 
factor for the CONCERN press (pilot detection time) and the TOGA press (pilot reaction time), 
respectively. The pilots exhibited an (averaged) increased response mean time of 3.6 seconds for 
the CONCERN press and 5.2 seconds for the TOGA press with the Embedded Conventional 
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display concept as compared to the other three display concepts. 



Figure 17. Autopilot oscillation scenario mean CONCERN press times per display concept. 



Figure 18. Autopilot oscillation scenario mean TOGA press times per display concept. 
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The autopilot oscillations began at 580 feet AGL altitude on final approach, reaching a 1 dot 
deviation in glideslope at threshold. The inference from these results is that the pilots were able 
to detect autopilot oscillations more readily (by an average of 3.6 seconds) with the CH, EH, and 
PV display concepts as compared to the EC concept. The head-down PY concept had a 62° 
horizontal FOV; while the head-up view of the CH and EH concepts had a 40° horizontal FOV. 
The EC concept’s “minified” PFD display (compressed 40° horizontal FOV) probably caused the 
increased pilot detection time of the autopilot oscillations; thus, decreasing pilot recovery 
initiation by an average of 5.2 seconds (corresponding to 60 ft relative altitude difference) with 
the EC display when compared to the CH, EH, and PV displays. Small deviations from path are 
less apparent in the compressed 40° horizontal FOV of the EC concept. These display 
differences are significant, both statistically and in real-world flight operations, during the 
landing phase of flight where critical flight decisions are heavily constrained by time. 


Navigation System Error (small) 

Repeated measures ANOVA analyses (appendix A. 18 - A. 19) on the CONCERN button press 
time (pilot detection time) (F(3,21) = 24.17, p < .000) and the TOGA button press time (pilot 
reaction time) (F(3,21) = 14.51, p < .000) showed highly significant differences among the 
displays factor. Figures 19 and 20 graphically present the results of the Navigation System Error 
Scenario with the 75 foot discrepancy between sensor and inertial -referenced runway centerlines. 
For this scenario, the lateral position error was implemented at the beginning of the mn so the 
CONCERN and TOGA button press time measurements began at this point as well. Post hoc 
analysis of the CONCERN press time means (using the SNK test at a significance level of .05) 
showed no differences between the PV and EC displays and no differences between the CH and 
EH displays, but differences were noted between these two groups. The mean CONCERN press 
time (or pilot detection time) for the PV display was 5 seconds sooner than the EC concept and 
30 seconds sooner than the EH and CH concepts. These time differences correspond to a relative 
altitude difference of approximately 68 feet between the PV and EC concepts and 373 feet 
between the PV and EH/CH concepts for recognizing the small navigation system error. Post 
HOC comparisons of the TOGA press time means showed significant differences between the 
Pictorial Vision Concept and the other three concepts (CH, EH, and EC), but no differences 
between the latter three concepts. The mean TOGA press time (or pilot reaction time) for the PV 
display was 12 seconds sooner (corresponding to 146 ft relative altitude difference) than the EC 
concept and 23 seconds sooner (corresponding to 274 ft relative altitude difference) than the EH 
and CH concepts. 


30 



155 



Dis play Concept 

Figure 19. Small navigation system error scenario mean CONCERN press times per display 
concept. 
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Figure 20. Small navigation system error scenario mean TOGA press times per display concept. 


The inference from these results is that the Pictorial Vision display concept provided the 
pilots with better awareness of erroneous navigation system information than did the other 
display concepts. The 62° horizontal FOV of the PV display (as compared to the 40° horizontal 
FOV of the EH and EC displays) probably enabled pilot detection of the abnormal flight 
condition earlier and at higher altitudes (a 30-second-earlier detection time translates into 373 ft 
of altitude separation), indicating increased pilot awareness of the situation. Pilot reaction time 
(in the fomt of a TOGA press) to discontinue autoland approaches was considerably better with 
the Pictorial Vision display concept over the other three concepts (a 23-second earlier reaction 
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translates into 124 ft of altitude separation), again indicating increased pilot situation awareness 
probably due to the larger FOV of the PV display concept. 


Navigation System Error (large) 

Graphical inspection of the data showed one outlier (3-0 away from the grand mean) for the 
CONCERN press (pilot detection) time, resulting in unequal display variances. Removal of this 
outlier in the CONCERN press data reduced the affected display’s variance by a factor of four. 
(See fig. 21.) A repeated measures ANOVA analysis (appendix A.22) (excluding the pilot with 
the outlier) on the CONCERN button press detection time showed a highly significant main 
effect for displays (F(3,18) = 14.85, p < .000). Significance of the sphericity test (p > 0.05) on 
the main factor, displays, indicated that the repeated measures ANOVA results should be 
accepted. Figure 22 graphically presents the results of the Navigation System Error Scenario 
with the 300 foot discrepancy between sensor and inertial-referenced runway centerlines. For 
this scenario, the lateral position error was implemented at the beginning of the mn so the 
CONCERN and TOGA button press time measurements began at this point as well. Post hoc 
comparisons of the CONCERN time means using SNK test at a 5-percent significance level 
showed no differences between the EC and PV concepts or the CFI and EH concepts; however, 
differences were noted between these two groups. The mean CONCERN press time (pilot 
detection time) for the EC/PV displays was 13 seconds sooner than the CH/EH displays, 
resulting in a 169 feet relative altitude separation between the two groups. 



Figure 21. Large navigation system error scenario CONCERN press display variances 
with and without inclusion of an outlier. 
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Figure 22. Large navigation system error scenario mean CONCERN press times per display 
concept. 


The inference from these results is that the Pictorial Vision and Embedded Conventional 
display concepts provided the pilots with better awareness of inaccurate navigation system 
information than did the Conventional HUD and Enhanced HUD display concepts. The head- 
down-only scan of the PV/EC displays (as opposed to the dual head-down/head-up scan with the 
CH/EH displays) probably enabled pilot detection of the abnormal flight condition earlier and at 
higher altitudes (a 13-second-earlier detection time translates into 169 ft of altitude separation), 
indicating increased pilot awareness of the situation. The magenta runway outline for the sensor 
in the multi-colored PV/EC displays probably helped improve pilot detection time with these 
displays as compared to the green outlines of the sensor and inertial-based mnways in the 
monochrome CH/EH displays. 


Freezing/Probes Scenarios 


Blanking Scenarios 

These are the only scenarios where the pilot was required to take control of the airplane and 
continue flying the approach. The blanking/back-up instrument time was approximately 20 
seconds for each scenario. At blanking, the turbulence was turned off but the wind remained on 
so that the plane maintained it’s inertial track until the pilot provided manual control input. This 
was done in order to maintain initial conditions for each of the display conditions for this 
scenario. For each blanking scenario, a repeated measures ANOVA on displays (only factor 
present) was perfomied for the following performance measures: 
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- vertical (altitude) path error rms, mean, and standard deviation 

- lateral path error mis, mean, and standard deviation 

- distance from path mis, mean, and standard deviation 


No differences (at the .01 significance level) were found among the four display concepts in 
the 27 repeated measures ANOVA analyses that were performed. (See appendix B.) Trends that 
were seen in the data are discussed in the remainder of this section. Lateral and vertical rms path 
errors, collapsed across blanking scenarios and pilots, showed that the sensor-based display 
concepts offered performance improvements over the Conventional HUD display concept. For 
Blanking Scenario 1 (blanks before turn to final), although not statistically significant, 
improvement in lateral and vertical mis path errors was observed in the sensor-based displays. 
(See fig. 23.) For Blanking Scenario 2 (blanks before roll-out on final), the vertical rms path 
errors remained relatively small and constant across the four displays; while the Embedded 
Conventional display had lower lateral RMS path errors as compared to the other three displays. 
(See fig. 24.) All eight pilots correctly initiated the commanded TCAS maneuver (descent) for 
Blanking Scenario 3 (blanks after TCAS resolution advisory on base) when using the three 
display concepts utilizing sensor infomiation. One pilot incorrectly initiated the commanded 
TCAS maneuver while flying the Conventional HUD display concept; while the other seven 
pilots initiated the correct response for the CH display concept. 



CH EH EC PV 

Dis play Concept 


Figure 23. Blanking scenario one-vertical and lateral mis path errors per display concept. 
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Figure 24. Blanking scenario two-vertical and lateral rms path errors per display concept. 


The blanking scenarios yielded no relevant information about spatial awareness differences 
between the display concepts being examined. This type of SA assessment technique, under this 
experiment design, does not appear to provide a useful measure of pilots’ situation awareness 
when comparing display concepts that incorporate sensor-based wire-frame runway and icon 
obstacles (Enhanced HUD, Embedded Conventional, Pictorial Vision) under IMC with a 
Conventional HUD display concept under VMC during autoland approaches. 


Probe Scenario 

For each Display Concept, the probe scenario occurred twice. Each pilot experienced the 
Probe scenario only once during the data collection mns. Figure 25 graphically shows the results 
for the small sampling of pilots in the Probe scenario. When using the EC display all three 
abnormal flight conditions (Flight Director (FD) Conflict with Autopilot (AP), TCAS Traffic 
Advisory, and Loss of Glideslope) were detected and correctly identified for 5/6 occurrences. 
With the remaining three display concepts, the pilots detected and correctly identified 4/6 
occurrences of the abnomial flight conditions. The Conventional FIUD concept had one pilot 
missing the FD conflict with AP and the other pilot missing the TA. The Enhanced HUD 
concept had one pilot correctly identifying all 3 abnomial flight conditions and the other pilot 
missing the TA and the FD conflict with AP. The Pictorial Vision concept had both pilots 
missing the FD conflict with AP. No pilot missed the loss of glideslope flight condition. 

It is suggestive from these results that the EC concept provided somewhat better situational 
awareness than the other three displays. The greater number of detections with the EC concept 
was probably attained because the pilot’s visual scan was contained in a smaller, centrally- 
located area with this concept as compared with the other three display concepts. 
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Figure 25. Probe scenario number of detections of abnormal flight conditions per display 
concept. 


Subjective Data Analyses 


Six questionnaires (one for each display concept, one comparing the three display concepts 
flown in IMC, and one for the probe scenario) were administered to each pilot during this 
simulation experiment. (See the right hand portion of table 3 in the Procedures section- 
Organization of Trials. Also reference appendices C and D. ) In addition to the display-specific 
questions, the pilots were asked to assign a modified C-H rating after being exposed to a block of 
scenarios for a specific display concept. 


Display Questionnaire Ratings 

A repeated measures ANOVA analyses on the modified C-H operator ratings showed a highly 
significant effect (F(3,21) = 4.964, p < .009) for the factor displays. Post hoc analysis of the 
modified C-H ratings using SNK test with significance level .05 showed differences between the 
PV display concept and the other three display concepts (CH, EH and EC). The PY display 
concept had a lower operator demand level and difficulty level than the other three displays. No 
significant differences were seen between the means of CH, EH, and EC display concepts. 
Figure 26 graphically shows the mean modified C-H ratings for each display concept. The PV 
display concept had a mean modified C-H rating of 1.625 which indicated a “easy, desirable” 
difficulty level to the pilot. A modified C-H rating in this range indicates that the “operator 
mental effort is low and desired performance is adequate.” The mean modified C-H ratings for 
the remaining three display concepts indicated a “fair, mildly” difficulty level to the pilot and 
that “acceptable operator mental effort is required to attain adequate system performance” by the 
pilot. 
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Figure 26. Modified Cooper-Harper mean ratings for each display concept. 

Individual Display Concept Questionnaire 

A series of repeated measures ANOVAs were performed on the pilot responses to individual 
display questionnaire inquires about the following (See appendix D): 

1 . encountering the anomalous cues/detection time scenarios 

2. monitoring an autoland approach 

3. pilot workload in monitoring autopilot functions, airborne traffic and airport surface 
traffic 

4. ease of making the decision to go around 

5. ease of maintaining situational awareness 


The anomalous cues/detection time scenarios encountered by each pilot are as follows: 

1 . Flight Director conflict with Autopilot 

2. Flight Director conflict with Raw Data 

3. Aircraft Incursion on Final 

4. Flag TOGA (localizer failure) 

5. Autopilot Oscillation 

6. Navigation System Error - small (75 feet)/large (300 feet) 

For each anomalous cues/detection time scenario, the pilots rated the Pictorial Vision concept 
as the easiest display concept among the three displays flown in IMC to detect and understand 
the unusual flight condition. The pilots ranked the Enhanced HUD system second of the three 
displays flown in IMC for ease of detecting and understanding all the anomalous cues/detection 
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time scenarios, except for two instances. The Embedded Conventional concept was ranked 
second for ease in understanding what was wrong in the scenario in which the Flight Director and 
the Flight Director Raw Data are in conflict and for ease in understanding what was wrong in the 
scenario in which a Failure Flag (loss of localizer signal) occurred. No statistical differences 
between the means of the PV concept and the EH concept were found in any of the anomalous 
cues/detection time scenarios except in the scenario of recognizing an aircraft runway incursion. 
For this scenario, pilot opinion inferred that the Pictorial Vision concept was very easy in 
recognizing an aircraft incursion; while, the Enhanced HUD concept neither aided nor hindered 
the recognition of an aircraft incursion. The pilots ranked the Embedded Conventional concept 
last among the three displays flown in IMC for ease of detecting and understanding all of the 
anomalous cues/detection time scenarios, except for the two conditions noted above. This low 
ranking can probably be attributed to the difficulty of recognizing an abnormal condition in the 
EC’s compressed 40° field of view as presented in the PFD as compared to the 40° FOV found in 
the EH’s out-the-world scene or the 62° FOV found in the PV’s synthetic scene. 

Out of the four display concepts flown, the pilots ranked the Pictorial Vision concept as the 
display that was easiest to use to maintain situational awareness, to monitor an autoland 
approach, to interpret infomiation from the mnway and obstacle detecting sensor systems, and to 
make the decision to go around. The PV concept was also the pilot preference among the fours 
displays evaluated for ease of monitoring autopilot functions, airborne traffic and airport surface 
traffic. 


Final Questionnaire 

After completing all the mns for each Display Concept, the probe questionnaire, and the 
individual display concept questionnaires, the pilots completed a final questionnaire that 
involved detailed comparisons of the three display concepts (EH, EC, and PV) flown in IMC. 
Similar to the subjective results found for the individual display questionnaires, the Pictorial 
Vision concept was significantly preferred over the Enhanced HUD and the Embedded 
Conventional display concepts when considering all monitoring tasks (i.e., the approach, 
verifying the location of the mnway, detecting ground runway incursions, etc.). For each 
anomalous cues/detection time scenario, the Pictorial Vision concept had the highest pilot 
ranking among the three display concepts flown in IMC for ease of detecting and understanding 
the unusual flight condition. The pilots rated the Enhanced HUD concept second and the 
Embedded Conventional concept third of the three displays for ease of detecting and 
understanding the anomalous cues/detection time scenarios. The EC’s low ranking can probably 
be attributed to the difficulty of recognizing an abnormal flight condition in this concept’s 
“minified FOV” PFD display. 

Figure 27 shows the results of comparative rank ordering (of the display concepts flown in 
IMC) by the pilots for several categories on a scale of 1 (the least desirable display) to 10 (the 
most desirable display). For each category, the mean, maximum, and minimum pilot rankings 
(not plus or minus the standard deviations) for all experiment scenarios are presented. The pilots 
ranked each display’s effectiveness for allowing the pilot to monitor all flight tasks, maintain 
situation awareness and monitor an autoland in restricted visibility conditions, for reducing their 
overall workload, and for presenting them sensor-based information. Based on subjective 
rankings, the PV display concept offered considerable improvements in situation awareness and 
reductions in overall pilot workload (in both mean ranking and spread) over the EH and EC 
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display concepts. The pilot preference for the PV display concept over the other two display 
concepts may have been influenced by the reduced scan-time and larger field of view associated 
with this display. When using the Pictorial Vision concept, the pilots scan time was not divided 
between the Head-up and Head-down displays like it was in the Enhanced HUD concept, and the 
pilots had a much larger FOV (62° as compared to a compressed 40°) to determine unusual flight 
conditions than they did with the Embedded Conventional concept. 
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Figure 27. Comparative rank ordering by pilots for five categories. 
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Probe Questionnaire 


Several pilots commented that the intrusive SA assessment technique of the Probe scenario 
helped them remain vigilant while monitoring the autoland operations. After encountering this 
“surprise" scenario, the pilots commented that they were expecting more “surprises" which 
encouraged them to remain alert. However, no significant display differences were noted. 


Pilot Comments 

hi addition to the questionnaire results, pilots were also give the opportunity to provide 
comments on the questionnaires about the display concepts. Some of the more notable 
comments are the following: 

“What appears to be complete situational awareness. The field of view is very 
good and traffic alerts and resolutions are noticed very quickly. It almost gives a 
three dimensional effect. The pathway presents a fulltime unconscious 
evaluation of LOC and G/S performance which reduces the amount of 
crosschecking required of the raw data. This reduction in workload really pays 
off when unexpected problems arise which demand additional attention. One 
can allocate that additional time while still subconsciously monitoring the 
approach. “ [comments on advanced pictorial display concept] 


“When comparing all three systems PV [advanced pictorial display concept] was 
only system that would make a CAT IIIc a comfortable approach.” 


“The sensor information is a must for any future aircraft instrumentation for 
increased flight safety first and restricted visibility conditions approaches and 
landings.” 


“Overall SA was greatly improved over conventional systems. Easier to monitor 
not only flight progress, but the complete flight environment. I believe this 
alone lends itself to a safer operation for both the user aircraft and airspace it’s 
operating in." [comments on advanced pictorial display concept] 
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CONCLUDING REMARKS 


A simulation study was conducted using eight commercial airline pilots, repeatedly flying 
MLS-type autoland approaches to closely spaced parallel runways, to assess pilots’ situation 
awareness and workload while monitoring autoland operations utilizing three advanced display 
concepts. These concepts incorporated sensor-based wire-frame runway and icon obstacles and 
were flown under Instrument Meteorological Conditions. A conventional EFIS -based display 
concept utilizing a HUD without any sensor-based information provided an experimental control 
condition for the simulation. This conventional EFIS-based/HUD display concept was simulated 
under Visual Meteorological Conditions instead of IMC because the pilot had to visually detect 
the runway and other traffic without the aid of a sensor. Comparisons were made between the 
conventional EFIS-based/HUD concept and the advanced display concepts to ensure that the 
advanced concepts did not degrade a pilot’s current level of SA or increase his or her workload 
during autoland operations. Various situational awareness measurement techniques, involving 
conflicting traffic situation assessments, main display failures, and conflicting position 
information, were used to assess the pilots’ situation awareness with the different display 
concepts, both objectively and subjectively. The situation awareness tools utilized in the 
experiment proved to be most effective in the assessment (with the exception of the Blanking 
Scenarios), in that the results were consistent across and within the objective and subjective 
measures. Two subjective measures, modified Cooper-Harper (C-H) ratings and questionnaire 
display rankings, were utilized in this experiment to assess the pilot’s mental workload while 
monitoring autoland operations with each of the display concepts. 

Objective data analyses for the Navigation System Error scenarios revealed that better 
situation awareness performance (quicker recognition and understanding of inaccurate navigation 
system information) was achieved with the Pictorial Vision display concept as compared to the 
other display concepts (Conventional HUD, Enhanced HUD and Embedded Conventional). 
Analyses of the data for the Autopilot Oscillations Scenario showed a dramatic reduction in 
situational awareness (significant both statistically and operationally) when using the Embedded 
Conventional display concept during landing. No significant differences existed among the other 
three display concepts for the Autopilot Oscillations scenario. The objective data analysis for the 
FLAG TOGA (loss of localizer signal) scenario yielded no significant differences between the 
three sensor-based displays, but there were differences noted between the Embedded 
Conventional and the Conventional HUD display concepts. Although statistically significant, 
these differences are not operationally significant for a transport aircraft. 

A summary of the numerous subjective results indicate a very strong preference for the 
Pictorial Vision display concept over the Enhanced HUD and Embedded Conventional display 
concepts when considering all monitoring tasks (i.e., the approach, verifying the mnway location, 
detecting ground mnway incursions, monitoring autopilot functions, monitoring airborne traffic, 
etc.). The major subjective results of the study showed that substantial improvements in 
situation awareness for detecting and understanding an abnormal flight condition (e.g., 
conflicting traffic, loss of localizer signal, etc.) were provided by the integrated pictorial concept 
(Pictorial Vision) when compared to the other two advanced sensor display concepts (Enhanced 
HUD and Embedded Conventional). In addition to increased situation awareness, subjective 
rankings indicated that the pictorial concept offered reductions in overall pilot workload (in both 
mean ranking and spread) over the two advanced sensor display concepts. Data analyses on 
modified C-H ratings of mental workload showed that tire Pictorial Vision concept had a lower 
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operator demand level and difficulty level than the other three display concepts. Out of the four 
display concepts flown, the pilots ranked the Pictorial Vision concept as the display that was 
easiest to use to maintain situational awareness, to monitor an autoland approach, to interpret 
information from the runway and obstacle detecting sensor systems, and to make the decision to 
go around. 

Both objective and subjective results indicate that an integrated pictorial display has shown 
significant promise for providing improved situational awareness, without increments in pilot 
workload, during autoland operations and should therefore be further studied with regard to 
potential corresponding safety benefits. This type of format is expected to provide the 
cornerstone for an effective synthetic vision system, a system which is believed to be an enabling 
technology for solving restricted visibility problems associated with advanced subsonic and 
future high speed civil transports. 
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Appendix A 


Repeated Measures ANOVA Summary Tables for Anomalous Cues/Detection Time 
Scenarios 


43 



A. 1 ANOVA Summary Table For CONCERN Press Time For Flight Director Conflict 
With Autopilot 


Source of Variation 

SS 

DF 

MS 

F 

iwiiiisii 

SUBJECTS 

588.44 

7 

84.06 



DISPFAY 

102.59 

3 

34.20 

1.46 

.253 

SUBJECT BY DISPFAY 

490.20 

21 

23.34 



TOTAF 

1181.23 

31 





A. 2 ANOVA Summary Table For TOGA Press Time For Flight Director Conflict With 
Autopilot 


Source of Variation 

SS 

DF 

MS 

F 

IMgBi iM 

SUBJECTS 

39223.45 

7 

5603.35 



DISPFAY 

273.05 

3 

91.02 

1.73 

.192 

SUBJECT BY DISPFAY 

1105.72 

21 

52.65 



TOTAF 

40602.22 

31 





A. 3 ANOVA Summary Table For Time Difference Between CONCERN And TOGA 
Presses For Flight Director Conflict With Autopilot 


Source of Variation 

SS 

DF 

MS 

F 

IMdElSiiB 

SUBJECTS 

31175.18 

7 

4453.60 



DISPFAY 

254.22 

3 

84.74 

1.43 

.261 

SUBJECT BY DISPFAY 

1240.88 

21 

59.09 



TOTAF 

32670.28 

31 





A.4 ANOVA Summary Table For CONCERN Press Time For Flight Director Conflict 
With Raw Data 


Source of Variation 

SS 

DF 

MS 

F 

Sig ofF 

SUBJECTS 

658.48 

7 

94.07 



DISPFAY 

45.19 

3 

15.06 

1.09 

.377 

SUBJECT BY DISPFAY 

291.22 

21 

13.87 



TOTAF 

994.89 

31 





A.5ANOVA Summary Table For TOGA Press Time For Flight Director Conflict With 
Raw Data 


Source of Variation 

SS 

DF 

MS 

F 

WdfliliJII 

SUBJECTS 

34541.07 

7 

4934.44 



DISPFAY 

37.63 

3 

12.54 

.26 

.852 

SUBJECT BY DISPFAY 

1007.01 

21 

47.95 



TOTAF 

35585.71 

31 
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A. 6 ANOVA Summary Table For Time Difference Between CONCERN And TOGA 
Presses For Flight Director Conflict With Raw Data 


Source of Variation 

SS 

DF 

MS 

F 

mmi-M 

SUBJECTS 

31357.58 

7 

4479.65 



DISPFAY 

90.06 

3 

30.02 

.78 

.517 

SUBJECT BY DISPFAY 

805.78 

21 

38.37 



TOTAF 

32253.42 

31 





A. 7 ANOVA Summary Table For CONCERN Press Time For Aircraft Incursion On 
Final (8F) 


Source of Variation 

SS 

DF 

MS 

F 

Sig of F 

SUBJECTS 

461.69 

7 

65.96 



DISPFAY 

425.47 

3 

141.82 

2.13 

.126 

SUBJECT BY DISPFAY 

1395.67 

21 

66.46 



TOTAF 

2282.83 

31 





A. 8 ANOVA Summary Table For TOGA Press Time For Aircraft Incursion On Final 
(8F) 


Source of Variation 

SS 

DF 

MS 

F 

Sig of F 

SUBJECTS 

14.94 

7 

2.13 



DISPFAY 

.34 

3 

.11 

.18 

.911 

SUBJECT BY DISPFAY 

13.57 

21 

.65 



TOTAF 

28.85 

31 





A. 9 ANOVA Summary Table For Time Difference Between CONCERN And TOGA 
Presses For Aircraft Incursion On Final (8F) 


Source of Variation 

SS 

DF 

MS 

F 

Sig of F 

SUBJECTS 

488.03 

7 

69.72 



DISPFAY 

444.10 

3 

148.03 

2.29 

.108 

SUBJECT BY DISPFAY 

1356.96 

21 

64.62 



TOTAF 

2289.09 

31 





A. 10 ANOVA Summary Table For CONCERN Press Time For Aircraft Incursion On 
Final (8R) 


Source of Variation 

SS 

DF 

MS 

F 

Sig of F 

SUBJECTS 

26.55 

7 

3.79 



DISPFAY 

12.68 

3 

4.23 

2.14 

.125 

SUBJECT BY DISPFAY 

41.40 

21 

1.97 



TOTAF 

80.63 

31 
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A. 1 1 ANOVA Summary Table For TOGA Press Time For Aircraft Incursion On Final 
(HR) 


Source of Variation 

ss 

DF 

MS 

F 

mmiiM 

SUBJECTS 

12.68 

7 

1.81 



DISPLAY 

.53 

3 

.18 

.29 

.831 

SUBJECT BY DISPLAY 

12.78 

21 

.61 



TOTAL 

25.99 

31 





A. 12 ANOVA Summary Table For Time Difference Between CONCERN And TOGA 
Presses For Aircraft Incursion On Final (8R) 


Source of Variation 

SS 

DL 

MS 

L 

iniisfua 

SUBJECTS 

28.99 

7 

4.14 



DISPLAY 

8.99 

3 

3.00 

1.58 

.224 

SUBJECT BY DISPLAY 

39.82 

21 

1.90 



TOTAL 

77.80 

31 





A. 13 ANOVA Summary Table For TOGA Press Time For Flag TOGA 


Source of Variation 

SS 

DF 

MS 

F 


SUBJECTS 

926.56 

7 

132.37 



DISPLAY 

152.45 

3 

50.82 

1.84 

.170 

SUBJECT BY DISPLAY 

578.60 

21 

27.55 



REP 

10.97 

1 

10.97 

1.45 

.267 

SUBJECT BY REP 

52.93 

7 

7.56 



DISPLAY BY REP 

62.98 

3 

20.99 

2.41 

.095 

SUBJECT BY DISPLAY BY REP 

182.58 

21 

8.69 



TOTAL 

1967.07 

63 





A. 14 ANOVA Summary Table For TOGA Press Time For Flag TOGA (excluding Pilot 
4) 


Source of Variation 

SS 

DF 

MS 

F 


SUBJECTS 

16.56 

6 

2.76 



DISPLAY 

25.49 

3 

8.50 

2.77 

.072 

SUBJECT BY DISPLAY 

55.28 

18 

3.07 



REP 

1.66 

1 

1.66 

.37 

.564 

SUBJECT BY REP 

26.75 

6 

4.46 



DISPLAY BY REP 

14.23 

3 

4.74 

1.86 

.173 

SUBJECT BY DISPLAY BY REP 

45.94 

18 

2.55 



TOTAL 

185.91 

55 





46 









































































































































A. 15 ANOVA Summary Table For CONCERN Press Time For Autopilot Oscillation 


MS 


16.52 


53.89 


6.78 


4.98 


3.61 


3.17 


4.87 


Source of Variation 

SS 

DF 

SUBJECTS 

115.66 

7 

DISPLAY 

161.68 

3 

SUBJECT BY DISPLAY 

142.39 

21 

REP 

4.98 

1 

SUBJECT BY REP 

25.25 

7 

DISPLAY BY REP 

9.50 

3 

SUBJECT BY DISPLAY BY REP 

102.21 

21 

TOTAL 

561.67 

63 



A. 16 ANOVA Summary Table For TOGA Press Time For Autopilot Oscillation 


MS 


66.48 


113.30 


6.37 


3.71 


1.41 


4.53 


2.69 


Source of Variation 

SS 

DF 

SUBJECTS 

465.33 

7 

DISPLAY 

339.90 

3 

SUBJECT BY DISPLAY 

133.67 

21 

REP 

3.71 

1 

SUBJECT BY REP 

9.84 

7 

DISPLAY BY REP 

13.59 

3 

SUBJECT BY DISPLAY BY REP 

56.45 

21 

TOTAL 

1022.49 

63 



A. 17 ANOVA Summary Table For Time Difference Between CONCERN And TOGA 


Source of Variation 

SS 

DF 

SUBJECT 

340.42 

7 

DISPLAY 

33.69 

3 

SUBJECT BY DISPLAY 

136.56 

21 

REP 

17.27 

1 

SUBJECT BY REP 

19.24 

7 

DISPLAY BY REP 

9.14 

3 

SUBJECT BY DISPLAY BY REP 

83.98 

21 

TOTAL 

640.30 

63 



MS 


48.63 


11.23 


6.50 


17.27 


2.75 


3.05 


4.00 


A. 18 ANOVA Summary Table For CONCERN Press Time For Small Navigation 
System Error 


MS 


75.18 


1982.16 


82.00 


Source of Variation 

SS 

DF 

SUBJECTS 

526.25 

7 

DISPLAY 

5946.49 

3 

SUBJECT BY DISPLAY 

1722.08 

21 

TOTAL 

8194.82 

31 




























































































































































A. 19 ANOVA Summary Table For TOGA Press Time For Small Navigation System 
Error 


Source of Variation 

ss 

DF 

MS 

F 

imuiBiiJi 

SUBJECTS 

2334.55 

7 

333.51 



DISPLAY 

2900.81 

3 

966.94 

14.51 

.000 

SUBJECT BY DISPLAY 

1399.87 

21 

66.66 



TOTAL 

6635.23 

31 





A. 20 ANOVA Summary Table For Time Difference Between CONCERN And TOGA 
Presses For Small Navigation System Error 


Source of Variation 

SS 

DF 

MS 

F 

gait* 

SUBJECTS 

1553.70 

7 

221.96 



DISPLAY 

1016.58 

3 

338.86 

3.97 

.022 

SUBJECT BY DISPLAY 

1791.55 

21 

85.31 



TOTAL 

4361.83 

31 





A. 21 ANOVA Summary Table For CONCERN Press Time For Large Navigation 
System Error 


Source of Variation 

SS 

DF 

MS 

F 

BdHSinl 

SUBJECTS 

576.26 

7 

82.32 



DISPLAY 

934.11 

3 

311.37 

3.76 

.026 

SUBJECT BY DISPLAY 

1737.21 

21 

82.72 



TOTAL 

3247.58 

31 





A. 22 ANOVA Summary Table For CONCERN Press Time For Large Navigation 
System Error (excluding pilot 8) 


Source of Variation 

SS 

DF 

MS 

F 

SigofF 

SUBJECTS 

187.23 

6 

31.21 



DISPLAY 

1144.00 

3 

381.33 

14.85 

.000 

SUBJECT BY DISPLAY 

462.32 

18 

25.68 



TOTAL 

1793.55 

27 





A. 23 ANOVA Summary Table For TOGA Press Time For Large Navigation System 
Error 


Source of Variation 

SS 

DF 

MS 

F 


SUBJECTS 

2056.94 

7 

293.85 



DISPLAY 

857.25 

3 

285.75 

2.35 

.102 

SUBJECT BY DISPLAY 

2553.53 

21 

121.60 



TOTAL 

5467.72 

31 
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A. 24 ANOVA Summary Table For Time Difference Between CONCERN And TOGA 
Presses For Lame Navigation System Error 


Source of Variation 

SS 

DF 

MS 

F 

jHISTiUdi 

SUBJECTS 

2065.16 

7 

295.02 



DISPLAY 

190.44 

3 

63.48 

1.63 

.214 

SUBJECT BY DISPLAY 

820.16 

21 

39.06 



TOTAL 

3075.76 

31 





A. 25 ANOVA Summary Table For CONCERN Press Time For Embedded TCAS 
Traffic Advisory 


Source of Variation 

SS 

DF 

MS 

F 

lidBiSiiii 

SUBJECTS 

2.20 

7 

.314 



DISPLAY 

1.208 

3 

.403 

1.062 

.383 

SUBJECT BY DISPLAY 

7.958 

21 

.379 



TOTAL 

11.366 

63 
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Appendix B 

Repeated Measures ANOVA Summary Tables for Blanking Scenarios 
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B.l ANOVA TABLE FOR VERTICAL RMS - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

IBM 

679.05 

7 

97.01 




1920.26 

3 

640.09 

1.83 

.174 

Subject by 
display 

7365.33 

21 

350.73 



Total 

9964.64 

31 





B.2 ANOVA TABLE FOR VERTICAL RMS - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


229.87 

7 

32.84 




10.03 

3 

3.34 

.07 

.976 

Subject by 
display 

1026.11 

21 

48.86 



Total 

1266.01 

31 





B.3 ANOVA TABLE FOR VERTICAL RMS - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


15147.37 

7 

2163.91 




1717.49 

3 

572.50 

.23 

.873 

Subject by 
display 

51707.57 

21 

2462.27 



Total 

68572.43 

31 





B.4 ANOVA TABLE FOR LATERAL RMS - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


8123.65 

7 

1160.52 




3223.98 

3 

1074.66 

.68 

.573 

Subject by 
display 

33099.54 

21 

1576.17 



Total 

44447.17 

31 
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B.5 ANOVA TABLE FOR LATERAL RMS - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

mmm 

1061.09 

7 

151.58 




523.90 

3 

174.63 

.13 

.942 

Subject by 
display 

28359.21 

21 

1350.44 



Total 

29944.20 

31 





B.6 ANOVA TABLE FOR LATERAL RMS - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


38631.88 

7 

5518.84 




3143.21 

3 

1047.74 

.19 

.905 

Subject by 
display 

118100.79 

21 

5623.85 



Total 

159875.88 

31 





B.7 ANOVA TABLE FOR DISTANCE FROM PATH RMS - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

nw 

7036.40 

7 

1005.20 




5328.19 

3 

1776.06 

.94 

.439 

Subject by 
display 

39637.70 

21 

1887.51 



Total 

52002.29 

31 





B.8 ANOVA TABLE FOR DISTANCE FROM PATH RMS - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


1096.47 

7 

156.64 



■■sett 

472.32 

3 

157.44 

.11 

.952 

Subject by 
display 

29387.01 

21 

1399.38 



Total 

30955.80 

31 
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B.9 ANOVA TABLE FOR DISTANCE FROM PATH RMS - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

ismBi 

50335.68 

7 

7190.81 




3891.43 

3 

1297.14 

.16 

.920 

Subject by 
display 

166797.42 

21 

7942.73 



Total 

221024.53 

31 





B.10 ANOVA TABLE FOR VERTICAL MEAN - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


647.88 

7 

92.55 




2741.23 

3 

913.74 

2.04 

.139 

Subject by 
display 

9410.65 

21 

448.13 



Total 

12799.76 

31 





B.ll ANOVA TABLE FOR VERTICAL MEAN - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


28.96 

7 

4.14 



USE 

12.59 

3 

4.20 

1.23 

.324 

Subject by 
display 

71.74 

21 

3.42 



Total 

113.29 

31 





B.12 ANOVA TABLE FOR VERTICAL MEAN - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


8791.43 

7 

1255.92 




767.31 

3 

255.77 

.13 

.944 

Subject by 
display 

42905.01 

21 

2043.10 



Total 

52463.75 

31 
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B.13 ANOVA TABLE FOR LATERAL MEAN - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

mmm 

1783.96 

7 

254.85 




2135.16 

3 

711.72 

.90 

.459 

Subject by 
display 

16652.94 

21 

793.00 



Total 

20572.06 

31 





B. 14 ANOVA TABLE FOR LATERAL MEAN - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


1645.20 

7 

235.03 




703.26 

3 

234.42 

.21 

.889 

Subject by 
display 

23565.40 

21 

1122.16 



Total 

25913.86 

31 





B.15 ANOVA TABLE FOR LATERAL MEAN - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


12008.89 

7 

1715.56 




1144.16 

3 

381.39 

.14 

.935 

Subject by 
display 

57376.21 

21 

2732.20 



Total 

70529.26 

31 





B.16 ANOVA TABLE FOR DISTANCE FROM PATH MEAN - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


2485.04 

7 

355.01 




5644.98 

3 

1881.66 

1.25 

.318 

Subject by 
display 

31676.01 

21 

1508.38 



Total 

39806.03 

31 
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B. 17 ANOVA TABLE FOR DISTANCE FROM PATH MEAN - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

ES9H 

1666.45 

7 

238.06 




690.65 

3 

230.22 

.20 

.896 

Subject by 
display 

24349.46 

21 

1159.50 



Total 

26706.56 

31 





B.18 ANOVA TABLE FOR DISTANCE FROM PATH MEAN - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


21413.99 

7 

3059.14 




2007.06 

3 

669.02 

.13 

.940 

Subject by 
display 

106126.26 

21 

5053.63 



Total 

129547.31 

31 





B.19 ANOVA TABLE FOR VERTICAL STD - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


166.93 

7 

23.85 




119.21 

3 

39.74 

2.02 

.142 

Subject by 
display 

412.96 

21 

19.66 



Total 

699.10 

31 





B.20 ANOVA TABLE FOR VERTICAL STD - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


210.37 

7 

30.05 




7.71 

3 

2.57 

.06 

.982 

Subject by 
display 

954.39 

21 

45.45 



Total 

1172.47 

31 
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B.21 ANOVA TABLE FOR VERTICAL STD - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

mmm 

7269.71 

7 

1038.53 




919.07 

3 

306.36 

.41 

.747 

Subject by 
display 

15647.45 

21 

745.12 



Total 

23836.23 

31 





B.22 ANOVA TABLE FOR LATERAL STD - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


6245.88 

7 

892.27 




1534.57 

3 

511.52 

.58 

.632 

Subject by 
display 

18401.93 

21 

876.28 



Total 

26182.38 

31 





B.23 ANOVA TABLE FOR LATERAL STD - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


1267.63 

7 

181.09 




288.58 

3 

96.19 

.24 

.864 

Subject by 
display 

8269.56 

21 

393.79 



Total 

9825.77 

31 





B.24 ANOVA TABLE FOR LATERAL STD - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


25179.60 

7 

3597.09 




2238.64 

3 

746.21 

.24 

.869 

Subject by 
display 

65835.80 

21 

3135.04 



Total 

93254.04 

31 
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B.25 ANOVA TABLE FOR DISTANCE FROM PATH STD - Blanking Scenario 1 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 

ES9H 

5128.69 

7 

732.67 




1106.86 

3 

368.95 

.61 

.613 

Subject by 
display 

12615.56 

21 

600.74 



Total 

18851.11 

31 





B.26 ANOVA TABLE FOR DISTANCE FROM PATH STD - Blanking Scenario 2 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


1429.35 

7 

204.19 




319.94 

3 

106.65 

.26 

.854 

Subject by 
display 

8643.77 

21 

411.61 



Total 

10393.06 

31 





B.27 ANOVA TABLE FOR DISTANCE FROM PATH STD - Blanking Scenario 3 


Source of 
Variation 

SS 

DF 

MS 

F 

Sig of F 


28123.42 

7 

4017.63 




2330.82 

3 

776.94 

.22 

.879 

Subject by 
display 

72897.44 

21 

3471.31 



Total 

103351.68 

31 
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Appendix C 

Sample Questionnaires - Display, Probe, and Final 


58 



N ame : 

Date: 

Display Evaluation 


(check one) 

Conventional HUD (CH) 

Enhanced HUD (EH) 

Embedded Conventional (EC) 

Pictorial Vision (PV) 

1. The display you have been monitoring is now to be evaluated over all of the 
experimental conditions encountered by using the modified Cooper-Harper rating scale 
shown below. In determining the Cooper-Harper operator rating for this display, it is 
important to start at the "operator decisions" block located at the lower left of the chart. 



Cooper-Harper Rating 


59 









2 . 


In the previous question, you determined the Cooper-Harper operator rating number for 
this display. When you selected the Cooper-Harper rating number among the final group 
of three (with the exception of a Cooper-Harper rating of 10 ), what adjective description 
listed under "Difficulty Level/Operator Demand Level" influenced your decision? Please 
discuss. (Example: I selected a rating of 3 because the symbology layout provided ...) 


In using this display: 

3. Evaluate the ease of monitoring an autoland approach. 


very 

somewhat 

neutral 

somewhat 

very 

hard 

hard 


easy 

easy 







4. Evaluate the ease in interpreting the information from the runway and obstacle detecting 
sensor systems in this display concept. 


very 

somewhat 

neutral 

somewhat 

very 

hard 

hard 


easy 

easy 




□ 
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5. 


Evaluate the ease of recognizing an aircraft runway incursion. 


very somewhat neutral somewhat very 

hai'd hai'd easy easy 


6. For the scenario in which the autopilot stops following the Flight Director commands, 
evaluate: 

A. the ease in detecting that something was wrong. 

very somewhat neutral somewhat very 

hai'd hai'd easy easy 

□ □ □ □ □ 


B. the ease in understanding what was wrong. 

very somewhat neutral somewhat very 

hai'd hai'd easy easy 

□ □ □ □ □ 


7. For the scenario in which the Flight Director and the F/D Raw Data are in conflict, 
evaluate: 

A. the ease in detecting that something was wrong. 

very somewhat neutral somewhat very 

hard hard easy easy 

□ □ □ □ □ 


B. the ease in understanding what was wrong. 

very somewhat neutral somewhat very 

hard hard easy easy 
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8. For the scenario in which the autopilot (assume a navigational system error) and the 
sensor data were in conflict, evaluate: 


A. the ease in detecting that something was wrong. 


very 

somewhat 

neutral 

somewhat 

very 

hai’d 

hai’d 


easy 

easy 




□ 



B. the ease in understanding what was wrong. 


very 

somewhat 

neutral 

somewhat 

very 

hai’d 

hard 


easy 

easy 




□ 



9. For the scenario in which an autopilot oscillation was encountered, evaluate: 
A. the ease in detecting that something was wrong. 


very 

somewhat 

neutral 

somewhat 

very 

hard 

hard 


easy 

easy 




□ 



B. the ease in understanding what was wrong. 


very 

somewhat 

neutral 

somewhat 

very 

hard 

hard 


easy 

easy 




□ 
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10. For the scenario in which a Failure Flag occurred, evaluate: 


A. the ease in detecting that something was wrong. 

very somewhat neutral somewhat very 

hai’d hai’d easy easy 

□ □ □ □ □ 


B. the ease in understanding what was wrong. 

very somewhat neutral somewhat very 

hai’d hai’d easy easy 

□ □ □ □ □ 

11. Rate your overall "workload". 

very somewhat neutral somewhat very 

hai’d hai’d easy easy 

□ □ □ □ □ 


12. Rate your workload in monitoring only the autopilot functions, i.e. course errors, 
airspeed errors, etc. 

very somewhat neutral somewhat very 

hard hard easy easy 


13. Rate your workload in monitoring only airborne traffic. 

very somewhat neutral somewhat very 

hard hard easy easy 
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14. Rate your workload in monitoring only airport surface traffic. 


very 

somewhat 

neutral 

somewhat 

very 

hai'd 

hai'd 


easy 

easy 







15. Evaluate the ease of making the decision to go-around. 


very 

somewhat 

neutral 

somewhat 

very 

hai'd 

hai'd 


easy 

easy 







16. Evaluate the ease of maintaining situation awareness. (Note: One could define situation 
awareness as "...the pilot has an integrated understanding of the factors that will 
contribute to the safe flying of the aircraft under normal or non-normal conditions". 

very somewhat neutral somewhat very 

hard hard easy easy 

□ □ □ □ □ 


17. How would you assign crew role/duties in using this system? 
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18 Please discuss the advantages of this display concept: 


19. Please discuss the disadvantages of this display concept: 



20. What improvements would you suggest for this display system? 
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21. Was there any symbology in this display that either appeared confusing or should be 
changed? 


22. Suggestions/comments: 
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NAME: 

DATE: 


PROBE QUESTIONNAIRE 


(check one) 

Conventional HUD (CH) 

Enhanced HUD (EH) 

Embedded Conventional (EC) 

Pictorial Vision (PV) 

The experimenter will complete the probe questionnaire based upon the verbal comments obtained from the 
evaluation pilot. This shall be done immediately after the blanking of the screen. 

Question: 

1 . At the moment the screen went blank, what was your overall situation? 


2. Have the pilot indicate his/her position on the appropriate approach plate the instant the screen 
went blank. 


3. At the moment the screen went blank, you were: 

right of path 

left of path 

on path 

4. At the moment the screen went blank, you were: 

above the glide slope 

below the glideslope 

on the glideslope 
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5. 


PICTORIAL DISPLAY: 


Have the pilot indicate his/her position with respect to the cross-sectional drawing of the 
pathway. 


6. CONVENTIONAL AND PICTORIAL 

a. Your lateral error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

b. Your vertical error was : 0 - 1/4 - 1/2 - 3/4 -1-1 1/2-2 dot error units 


7. Using the same approach plate in question two, have the pilot indicate the location of 
any target aircraft along with their TCAS symbology status. 

8. Mention any other anomalies you may have detected. 


9. If this had been an actual flight situation, what would you have done? (prioritize) 
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10. Evaluate this display for ease of maintaining "spatial awareness" while monitoring the approach. 


very 

somewhat 

neutral 

somewhat 

very 

hard 

hard 


easy 

easy 







11. Suggestions and/or comments concerning the probe scenario. 




N ame : 

Date: 

Final Questionnaire 

1. Based upon exposure to the three cockpit displays flown during this experiment in IMC 
conditions, please indicate your OVERALL relative ranking/grading of the displays by placing 
"EH", "EC" ,or "PV" at the appropriate location on the horizontal line shown below. In 
determining the overall evaluation, consider all monitoring tasks, i.e., the approach, verifying 
the location of the runway, detecting ground runway incursions, etc. 

where: 

"EH": indicates Enhanced HUD (IMC) 

"EC": indicates Embedded Conventional (HUD info in PD) 

"PV": indicates Pictorial Vision 


EXAMPLE 
1 1 1 

1 

1 

PV 

1 

EH 

EC 

1 1 


1 1 1 

1 2 
least desirable 
display 

1 1 1 

3 

4 

1 

5 

1 

6 

1 

7 

1 1 

8 

9 10 

most desirable 
display 

1 1 1 

1 2 
least desirable 
display 

3 

4 

5 

6 

7 

8 

9 10 

most desirable 
display 


2. Discuss your rationale in determining the overall evaluation in item 1 above. 


(Continue on next page) 
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Raiik/Grade the three displays flown in IMC conditions with respect to the following factors. 


where: 

"EH": indicates Enhanced HUD (IMC) 

"EC": indicates Embedded Conventional (HUD info in PD) 
"PV": indicates Pictorial Vision 

3. improving flying safety 


1234 567 89 10 

least desirable most desirable 

display display 

4. monitoring an "autoland" in restricted visibility conditions 


1234 567 89 10 

least desirable most desirable 

display display 
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5. 


monitoring other traffic 


1 2 3 4 5 6 7 8 

least desirable 
display 

9 10 

most desirable 
display 

6. allowing you to think ahead of the aircraft 

1 1 1 1 1 1 1 1 

1 1 1 

1 2 3 4 5 6 7 8 

least desirable 
display 

9 10 

most desirable 
display 

7. presenting sensor-based information 

1 1 1 1 1 1 1 1 

1 1 1 

1 2 3 4 5 6 7 8 

least desirable 
display 

9 10 

most desirable 
display 

8. recognizing aircraft runway incursions 

1 1 1 1 1 1 1 1 

1 1 1 

1 2 3 4 5 6 7 8 

least desirable 
display 

9 10 

most desirable 
display 

9. detecting a Flight Director and F/D Raw Data conflict. 

1 1 1 1 1 1 1 1 1 1 1 

1 2 3 4 5 6 7 8 

least desirable 
display 

9 10 

most desirable 
display 

10. understanding a Flight Director and F/D Raw Data conflict. 

1 1 1 1 1 1 1 1 1 1 1 

1 2 3 4 5 6 7 8 

least desirable 
display 

9 10 

most desirable 
display 

1 1 . detecting the scenario where the autopilot stops following the F/D 

1 1 1 1 1 1 1 1 1 1 1 

1 2 3 4 5 6 7 8 

least desirable 
display 

9 10 

most desirable 
display 
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12. understanding the scenario where the autopilot stops following the F/D 


1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

13. detecting an autopilot (assume a navigational system error) and sensor conflict. 

1 1 1 1 1 1 1 1 1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

14. understanding an autopilot (assume a navigational system error) and sensor conflict. 

1 1 1 1 1 1 1 1 1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

15. detecting an autopilot oscillation. 

1 1 1 1 1 1 1 

1 

1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

16. understanding an autopilot oscillation. 

1 1 1 1 1 1 1 

1 

1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

17. making the decision to go-around 

1 1 1 1 1 1 1 

1 

1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

18. reducing overall workload 

1 1 1 1 1 1 1 

1 

1 1 1 


1234 567 89 10 

least desirable most desirable 

display display 
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19. maintaining situation awareness* 


1234 567 89 10 

least desirable most desirable 

display display 

Note: One could define situation awareness as "...the pilot has an integrated understanding of 
the factors that will contribute to the safe flying of the aircraft under normal or non-normal 
conditions." 


20. The Enhanced HUD (EH) system is a useful concept. 


strongly 

disagree 

neutral 

agree 

strongly 

disagree 




agree 

The Embedded Conventional (EC) system is 

a useful concept, 

strongly 

disagree 

neutral 

agree 

strongly 

disagree 




agree 

□ 

□ 

□ 

□ 

□ 


22. The Pictorial Vision (PV) system is a useful concept. 

strongly disagree neutral agree strongly 
disagree agree 


23. I would feel comfortable flying with the sensor information presented in the 
Enhanced HUD (EH) concept. 

strongly disagree neutral agree strongly 
disagree agree 
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24. I would feel comfortable flying with the sensor information presented in the 
Embedded Conventional (EC) concept. 

strongly disagree neutral agree strongly 

disagree agree 

□ □ □ □ □ 

25. I would feel comfortable flying with the sensor information presented in the 
Pictorial Vision (PV) concept. 

strongly disagree neutral agree strongly 

disagree agree 


26. The sensor information is needed for landing in restricted visibility conditions. 

strongly disagree neutral agree strongly 
disagree agree 


27. The sensor information could increase flight safety. 

strongly disagree neutral agree strongly 
disagree agree 


28. Comments/suggestions concerning this experiment: 
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Appendix D 

Captured Comments on Questionnaires 
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Pilot Comments on Individual Display Questionnaires 


Display Evaluation 

(check one) 

V Conventional HUD (CH) 

Enhanced HUD (EH) 

Embedded Conventional (EC) 

Pictorial Vision (PV) 

1. The display yon have been monitoring is now to be evaluated over all of the 
experimental conditions encountered by using the modified Cooper-Harper rating 
scale shown below. In determining the Cooper-Harper operator rating for this 
display, it is important to start at the "operator decisions" block located at the 
lower left of the chart. 



Cooper-Harper Rating Mean = 2.75. 
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2. In the previous question, you determined the Cooper- Harper operator rating 
number for this display. When you selected the Cooper- Harper rating number 
among the final group of three (with the exception of a Cooper-Harper rating of 
10), what adjective description listed under "Difficulty Level/Operator Demand 
Level" influenced your decision? Please discuss. (Example: I selected a rating of 
3 because the symbology layout provided ...) 

Pilot 1: #2 - easy/desirable. Superb piece of equipment - 1 wish all a/c were equipped 
with the HUD. 

Pilot 2: Rating 2. The VFR environment is easier than IFR. All cross checks are 
contained inside until the runway environment is in view. All symbols are easily 
identified. 

Pilot 3: The symbology and overall display is easy to read. All required information is 
easily attainable using the HUD. The HUD itself is a good tool in that it allows the pilot 
to read required info while maintaining visual contact with the outside world. 

Pilot 4: The major cues for loc/gs errors were the yellow flags on the ADI. I found 
myself crosschecking between the HUD and the EFIS instead of using only one area for 
all the cues. The EHSI gave me more field of view for TCAS traffic than on the HUD so 
once again I was crosschecking between the HUD and EFIS to get the big picture. 

Pilot 5: Level 3 was selected because the two displays HUD and EFIS giving acceptable 
performance required more mental workload due to monitoring both displays. While 
mental effort was acceptable it did border on level 4 adjective annoying. 

Pilot 6: Based on the fact that the symbology in both the HUD and EFIS displays are the 
same as the Enhanced HUD (EH) version, with the exception of the sensor generated 
images and being in sim VFR conditions, my written comments are the same as, and are 
reference the (EH) evaluation sheet. One slight difference is my scan pattern is divided 
70% HUD/outside and 30% EFIS/inside the cockpit. In the EH, the scan was opposite - 
30% HUD and 70 % EFIS. 

Pilot 7: Selected 4, mental workload in interpretation between HUD outside and lower 
screens. 

Pilot 8: I selected a rating of 2 because the symbology layout is compact while being 
clear and easily attainable. The biggest improvement to this display would be 
incorporation of sensor whereever possible. 

17. How would you assign crew role/duties in using this system? 

Pilot 1: PF: fly the a/c. PNF: monitor TCAS/warnings/comm/make callouts etc. As well 
as keep an outside traffic watch. 
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Pilot 2: Pilot Flying - Monitor LOC and GS and runway alignment and railway 
environment. Non-Flying Pilot - Monitor airspeed and make altitude callouts. 

Pilot 3: PF - 75% HUD 25% cockpit instruments. PNF - 25% HUD 75% cockpit 
instruments. 

Pilot 4: PNF accomplishes checklists. First pilot to notice anything abnormal calls out 
the abnormality. PF stays outside with HUD while PNF stays inside with EFIS. 

Pilot 5: As done currently pilot flying. Pilot not flying with more emphasis on 
monitoring HUD display. 

Pilot 6: Refer to EH eval. 

Pilot 7: Crew duties could remain normal. Pilot flying outside with crosscheck, pilot not 
flying callouts crosscheck inside. 

Pilot 8: Pilot flying would have full responsibility for the aircraft and approach to 
landing. PNF would be responsible for all checklists/call outs and traffic advisories - 
This is VMC conditions only. 

18 Please discuss the advantages of this display concept: 

Pilot 1: Helpful when the pilot is ‘outside’ the cockpit, that he can remain outside with 
all HUD info supplying the additional information. 

Pilot 2: The display in cockpit is easy to understand and the HUD provides redundant 
info and the runway environment is in view. 

Pilot 3: The ability to by looking outside while still obtaining required flight 
information. 

Pilot 4: HUD symbology allows PF the opportunity to watch outside while presenting 
flight path data that he would otherwise have to obtain by looking inside. Gives him a 
faster reaction time to respond to the changing conditions. 

Pilot 5: HUD display is relatively uncluttered and easy to see through. F/D a/p guidance 
easy to understand. Situational awareness ok not as good as pictorial display. EFIS 
display conventional and easy to use by itself, except for Flt/Dir lag on ADI. Both 
displays together cover most concerns. TCAS traffic and determining their relationship 
to aircraft was much easier than pictorial display. Also easier to determine their 
movement. 

Pilot 6: Refer to EH eval. 
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Pilot 7 : Keeps pilot flying outside with most needed info always available. Provide 
better time to make go around decision. 


Pilot 8: The major advantage is the HUD display. Even in VMC conditions aircraft 
systems and navigation data is easier to read and interpret. This concept provides better 
quantity and quality data through the HUD. 

19. Please discuss the disadvantages of this display concept: 

Pilot 1: none. 

Pilot 2: none. 

Pilot 3: I think with HUDs in general a pilot could become too reliant on the system 
(HUD) and have a tendency to ignore his conventional instruments. 

Pilot 4: Still requires some crosschecking between outside world and EFIS. 

Pilot 5: Both displays require a Heads up and down approach to flying which requires 
more mental workload. In some instances 1 display is better than other for various pails 
of the approach (i.e., HUD is used more on short final). Concern some info such as 
loc/gs flags might be missed when using HUD. As there is a tendency to look through 
HUD possibly missing these indications on HUD as they don’t stand out as well. 

Pilot 6: Refer to EH eval. 

Pilot 7: Must crosscheck lower screen to confirm loss of Loc/GS. Also TCAS would 
only enter in with voice cautions or directives. The HUD wind vector arrow is in a 
different location (upper right) then the lower screen (lower It.) 

Pilot 8: Without the sensor data on the HUD crosschecking becomes slightly more 
difficult. 

20. What improvements would you suggest for this display system? 

Pilot 1: none. 

Pilot 2: none. 

Pilot 3: Possible moving airspeed, altitude off to each side in a vertical (tape) type 
format and maybe incorporate an ADA-slow/fast indicator. 

Pilot 4: Have gs and loc flags flash yellow when displayed. On HUD, instead of simply 
removing LOC/GS FD CMD bars, change the color of inop bar to flashing red before 
removing it or go from I to OFF in red on HUD. 
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Pilot 5: Get rid of one system. Use the HUD and a nav display or the EFIS ADI/HSI not 
both. Too much “split screen” scanning which increases workload. 

Pilot 6 : Refer to EH eval. 

Pilot 7 : Put wind arrow center of HUD, bottom where it would be more likely taken into 
scan as runway came into view. Now can be compared with airspeed/gspeed for possible 
wind delta or shear (low level). 

Pilot 8: None - sensor data incorporated on the HUD is displayed in another condition. 

21. Was there any symbology in this display that either appeared confusing or should be 
changed? 

Pilot 1: none. 

Pilot 2 : Change the size of the other aircraft in relation to their proximity to the aircraft 
(TCAS depiction). 

Pilot 3 : no comments given. 

Pilot 4: Consider not displaying any airborne traffic on the HUD unless it is an alert or 
resolution. Display all traffic on EHSI for planning purposes. 

Pilot 5: Altimeter and radar altimeter counters very difficult to read and interpret. Also 
get in way of HUD display when trying to see aircraft movement on airport. 

Pilot 6 : Refer to EH eval. 

Pilot 7 : Item 20. 

Pilot 8 : NO - all the symbology is easy to read, identify and interpret during normal and 
abnormal operations. 

22. Suggestions/comments: 


Pilot 1 : Love that HUD! 

Pilot 2 : none. 

Pilot 3 : Keep up the good work. 
Pilot 4: See above. 
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Pilot 5: Don’t think this combination of flight displays will work seem too cumbersome 
to fly. Suggest pictorial with change of TCAS traffic display to this display’s TCAS 
traffic symbology. 

Pilot 6 : Refer to EH eval. 

Pilot 7: no comments given. 

Pilot 8 : Combine sensor data on all HUD equipment. 
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Display Evaluation 


(check one) 


Conventional HUD (CH) 

✓ Enhanced HUD (EH) 

Embedded Conventional (EC) 

Pictorial Vision (PV) 


1. The display you have been monitoring is now to be evaluated over all of the 
experimental conditions encountered by using the modified Cooper-Harper rating 
scale shown below. In determining the Cooper-Harper operator rating for this 
display, it is important to stall at the "operator decisions" block located at the 
lower left of the chart. 



Cooper-Harper Rating Mean = 3,00. 
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2. In the previous question, you determined the Cooper- Harper operator rating 
number for this display. When you selected the Cooper- Harper rating number 
among the final group of three (with the exception of a Cooper-Harper rating of 
10), what adjective description listed under "Difficulty Level/Operator Demand 
Level" influenced your decision? Please discuss. (Example: I selected a rating of 
3 because the symbology layout provided ...) 

Pilot 1: symbology layout provided a very comprehensive ‘picture’ and certainly aids 
the pilot in the critical approach and landing phase of flight. 

Pilot 2 : I selected 3 because the HUD display is adequate to complete the task. However 
there is a great deal of information on the display, some of which may be redundant thus 
causing unnecessary clutter. 

Pilot 3 : “2” symbology and info available is good and for the most part easy to read and 
understand. 

Pilot 4 : A narrow field of view for traffic within HUD. Singular color features, which 
require further confirmation and additional crosschecking. No confirmation on HUD for 
GS and Loc capture. Large distances of eye travel between HUD and EFIS. 

Pilot 5 : 3 level selected because performance was adequate mental effort while 
acceptable was close to 4 level due to the annoying task of looking up and down between 
HUD and EFIS. 

Pilot 6: Because: (a) a cross check between the HUD and the EFIS display is used or 
desirable and (b) similar symbology having different meanings for flight and 
approach/navigational guidance is used in both the HUD and EFIS displays (i.e., Fit dir 
EFIS vs loc/gs HUD). 

Pilot 7: Selection 4. Although EHUD provides for increased field of view crosscheck 
between HUD display and lower screens requires more effort. 

Pilot 8: I selected a rating of 1 because the symbology of that display utilized the best 
aspects of all the displays including sensors. Even in heavy IMC situational awareness 
and traffic problems were easier to identify and deal with. Approach monitoring and 
possible runway incursions were very easy to recognize, as were instrumentation failures. 
IMC forces a better crosscheck. 

17. How would you assign crew role/duties in using this system? 

Pilot 1: P/F: in charge of “flying a/c”. Pilot not flying: conununication/monitoring 
instruments and proper callouts. 

Pilot 2: Pilot Flying: Monitor LOC and GS within parameters - monitor runway 
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environment. Non Flying Pilot - Monitor airspeed and altitude with call outs, monitor 
roll out on runway. 

Pilot 3 : PF 75% HUD 25% cockpit instruments. PNF 75% cockpit instruments 25% 
HUD. 

Pilot 4 : PF monitor HUD data. PNF monitor EFIS. 

Pilot 5: Pilot flying outside on HUD with pilot not flying monitoring inside with 
particular attention to NAV display. 

Pilot 6 : PF- always monitor a/c approach and navigation parameters using EFIS display. 
PNF - monitors airborne and gnd traffic using HUD while looking outside for the runway 
environment. PF - could transition to the HUD at 1000 or 500 feet until landing. 

Pilot 7 : After turn to final pilot flying monitors only EHUD display . Pilot not flying 
callouts and lower screens. 

Pilot 8 : PNF must do nothing but monitor aircraft instruments and performance while 
pilot flying runs checklist and communications. The main reason is to maintain 
situational awareness in case of a total loss of aircraft power (i.e. standby instruments, 
only IMC). 


18 Please discuss the advantages of this display concept: 

Pilot 1 : enhances total ‘picture’. ..makes for a very comprehensive approach and landing. 
Gives me the added information needed to make timely decision, but doesn’t give me too 
much which would clutter the instmments/HUD, etc. 

Pilot 2 : HUD display prepares the pilot for an easy transition to visual environment. 

Pilot 3 : PF - ability to obtain runway/airport visual cues sooner on an instrument 
approach while still monitoring flight information (loc, g/s, dh...). 

Pilot 4 : Provides PF with faster recognition of outside visual cues while maintaining 
some flight data. 

Pilot 5 : HUD gives a clear presentation (not too cluttered) on final approach segment 
with fairly easy flight path interpretation combining in with EFIS all info for monitoring 
is easily available for use. 

Pilot 6: Love to have a HUD in all my aircraft. Increased SA and reduced workload by 
being able to stay focused outside looking for the runway environment while still 
viewing/following a/c flight performance through the HUD. (ie) no need to keep a cross 
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check from inside the a/c to outside and back again. Reduced the possibility of inducing 
vertigo by eliminating head movements-therefore increases safety margins. 

Pilot 7 : Symbology using EHUD is easier to understand interpret. Also transition to 
outside during CAT II, or roll-out during CAT III is better. 

Pilot 8 : It incorporates sensors and all the best aspects of the other displays - HUD 
without the clutter. 

19. Please discuss the disadvantages of this display concept: 

Pilot 1 : none. 

Pilot 2 : It is difficult to recognize a failure in the G.S. or LOC. 

Pilot 3 : Could become too reliant on HUD and not crosscheck enough with cockpit 
instruments. 

Pilot 4 : I can compute colors faster than symbols, therefore, having all symbols in 
monocolor slows down my decision making process. 

Pilot 5 : Difficult to determine movement of traffic on ground. HUD - depth of field 
with TCAS traffic a little difficult to determine. Constant looking up and down from 
EFIS to HUD. Annoying. Occasional fixation on one display due to task at hand. 
Watching TCAS traffic easiest on NAV display. Making turn easiest on NAV display. 
Final approach segment easiest on HUD and failure flags easiest to notice on ADI. 

Pilot 6 : none 

Pilot 7 : Difficult to maintain crosscheck from EHUD to lower screens. Also TCAS 
icons on EHUD have no change in color only a change in shape, making the TA/RA 
more difficult to notice (i.e., must continue to crosscheck lower screens). 

Pilot 8 : NONE - in several ways its cleaner and not as cluttered as the others yet all the 
information is still available. 

20. What improvements would you suggest for this display system? 

Pilot 1 : none - 1 like what I see. 

Pilot 2 : A flashing LOC or GS sign in the event of failure should be displayed on the 
HUD screen. 

Pilot 3 : no comments given. 
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Pilot 4 : Add color. Remove TCAS displays on HUD unless alert/resolution. 

Pilot 5 : Remove altitude counters, in way of ground traffic. Hard to read - go to tapes. 
Change ground traffic symbology. Difficult to tell whether ground traffic is moving or 
aircraft display is moving. Combine any useful ADI info to HUD and do away with 
ADI. Use HUD and NAV display. 

Pilot 6 : see #22 below. Go-around decision will always depend on the situation that 
exists at any given moment (scenario). The goal is to make this decision as early as 
possible and is always going to be a personal choice based on comfort, experience and 
directives. 

Pilot 7 : In addition to the problem with TCAS icons needing color. Perhaps having the 
loc bar on EHUD flash instead just going away when LOC is lost. This would be more 
likely to be seen quickly. 

Pilot 8 : Improve the TCAS symbology on the HUD. 

21. Was there any symbology in this display that either appeared confusing or should be 
changed? 

Pilot 1 : no. 

Pilot 2 : The symbol for taxiing a/c should be closer to the actual size of the a/c relative 
to the taxiways and runway. 

Pilot 3 : The ground traffic symbology can be a little confusing and I think requires a 
little more attention than should be necessary. 

Pilot 4 : Add trend arrows to ground traffic. Flash LOC/GS upon capture. Flash FOC or 
GS before removing from HUD. 

Pilot 5 : See 20. 

Pilot 6 : Yes - loc/gs in HUD vs Fit Dir in the EFIS display same displayed symbology, 
different meanings. 

Pilot 7 : Item 20. 

Pilot 8 : The TCAS symbology on the HUD was not nearly as effective as the TCAS 
symbology on the HSI. 

22. Suggestions/comments: 

Pilot 1: none as of now. 
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Pilot 2 : The symbols for other a/c can become confusing with a lot of traffic (e.g. 

O’ Hare). If the symbols could be made relatively larger or smaller depending on their 
distance and/or altitude would help in identifying developing problems. 

Pilot 3 : no comments given. 

Pilot 4 : Provide wider field of view. 

Pilot 5 : Combine HUD ADI NAV display too hard to scan. Whole display should be 
combined or condensed for easier interpretation. 

Pilot 6 : Sensor data vs Nav error - if error is not large, then it’s hard to determine or 
even unable to determine until a confirmation exists at 300 feet when sensor blanking is 
switch off. Depending on the situation or approach - that could be much too late. A/P 
errors - again, should have redundant warnings for failure or when the a/p is unable to 
follow commanded guidance. 

Pilot 7 : Color was the drawback on EHUD. If color could be generated on screen all 
symbology would make more sense. 

Pilot 8 : All of the displays have pros and cons but the best combination would seem to 
be the HUD with sensors and bright clear graphics of heads down pictorial. 
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Display Evaluation 


(check one) 

Conventional HUD (CH) 

Enhanced HUD (EH) 

V Embedded Conventional (EC) 

Pictorial Vision (PV) 

1. The display yon have been monitoring is now to be evaluated over all of the 
experimental conditions encountered by using the modified Cooper-Harper rating 
scale shown below. In determining the Cooper-Harper operator rating for this 
display, it is important to stall at the "operator decisions" block located at the 
lower left of the chart. 



Cooper-Harper Rating Mean =3.25. 
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2. In the previous question, you determined the Cooper-Harper operator rating 
number for this display. When you selected the Cooper- Harper rating number 
among the final group of three (with the exception of a Cooper-Harper rating of 
10), what adjective description listed under "Difficulty Level/Operator Demand 
Level" influenced your decision? Please discuss. (Example: I selected a rating of 
3 because the symbology layout provided ...) 

Pilot 1 : no comments given. 

Pilot 2 : I selected a 2 rating because all information was displayed on the ADI and HSI. 
This reduced the crosschecking. However the transition to the outside environment is 
more difficult. 

Pilot 3 : “3” overall the information that is required is easily attainable and the 
symbology is good so long as the pilot is adequately trained and shown some of the 
anomalies of the system. 

Pilot 4 : Provided fairly good situational awareness along with normal performance cues 
without a great amount of eye travel during the crosscheck. 

Pilot 5 : 3 is selected because some symbology such as ground aircraft made mental 
effort a bit more difficult than low. System display performance was ok for monitoring 
of an autoland approach but was a little more difficult than other displays. 

Pilot 6 : Embedded symbology in the PFD is incorrectly sized for nearly all flight 
conditions, increasing pilot workload, rather than reducing it. Too much effort. 
Therefore, time, is used at critical phases of flight determining safe flight conditions 
(with regard to airborne and ground traffic/symbology). With respect to system 
malfunctions and monitoring flight parameters, certain symbology needs to be enhanced 
or enlarged (i.e., flight director). 

Pilot 7 : Selected 6. Due to display on NAV screen. The combination of screen size and 
number of and size of symbology displayed creates a high mental workload. The aircraft 
TCAS “staple” symbol needs to be a relative size to runway symbol. 

Pilot 8 : I selected a rating of 2 because the symbology layout provided easy monitoring 
of all aircraft functions except the detection of aircraft runway incursions. All other 
system monitoring for CAT III approaches was somewhat easy. Detection and decision 
making for all major discrepancies was also somewhat easy. 

17. How would you assign crew role/duties in using this system? 

Pilot 1 : PF: fly the a/c, make the decision regarding landing or go around, etc. PNF: 
monitor everything else, comm, make callouts, etc. Again, CAPT always makes the 
CAT 3 landings. 
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Pilot 2 : Flying Pilot monitor LOC and GS a/c on ground. Non Flying Pilot monitor 
airspeed call out altitudes. Both pilots monitor runway alignment and compare IRS 
image to sensor image. 

Pilot 3 : PF 75% cockpit 25% outside. PNF 75% outside 25% cockpit. 

Pilot 4 : PF primary area of concern would be to monitor the PD while the primary area 
of concern for the PNF would be to monitor the ND and forecast possible traffic 
problems. 

Pilot 5 : Pilot flying and non pilot flying would have to be inside monitoring until the 
alert height when the pilot flying would look outside to determine a landing would be 
made while the nonflying pilot would continue to monitor progress through the displays. 

Pilot 6 : PF - needs to concentrate all his/her efforts monitoring/following flight guidance 
information because of its reduced size and potential obscuration by the larger traffic 
symbology. PNF - back up flight guidance monitoring and reconflicting traffic 
info/symbology. 

Pilot 7 : Assume CAT IIIB, it would appear that the pilot flying concerned only with 
control of aircraft “monitoring”. Pilot not flying monitor traffic, TCAS, airport traffic, 
NAV deviations/errors. 

Pilot 8 : Copilot would fly the aircraft during the approach while the captain monitored 
the aircraft down to alert height where the decision to continue to land or go-around 
would be made on a CAT III approach. 

18 Please discuss the advantages of this display concept: 

Pilot 1 : Great to have it all superimposed in one small area. 

Pilot 2 : All information is presented on the panel instruments. All necessary data is 
presented on the ADI and HSI. 

Pilot 3 : Valuable information for low visibility approaches. 

Pilot 4 : Nice tight area for crosschecking data. I really like the traffic alert/resolution 
display on the ADI. It gives very good trend data in the vertical plane. 

Pilot 5 : All info is closely grouped for an easy instrument scan. Transitions from inside 
to outside would only occur once unlike a combined HUD display, which would be a 
continuous inside/outside transition. F/d a/p info easy to interpret on ADI. 

Pilot 6 : Obvious advantage is having all the information pertaining to total SA combined 
into one place - changing from a flight instrument crosscheck to an instrument stare. Go- 
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around decisions are more difficult for system problems due to the smaller symbology on 
the PFD. For ground traffic runway intrusions because symbology is too large and tends 
to obscure the runway and it’s environment. 

Pilot 7 : Runway sensor information is an advantage. However, the methods used to 
display misalign of magenta rwy and tracking are way too small. 

Pilot 8 : All primary data is located on two primary instruments - ADI and HSI. 
Symbology colors are excellent - bright and easy to recognize and evaluate. 

19. Please discuss the disadvantages of this display concept: 

Pilot 1 : Could possibly have too much info in the small area... but I think advantages 
outweigh the disadvantages. 

Pilot 2 : The obvious problem is the size of the display of the runway environment. As 
the FD and a/p line up with the runway symbol, the runway ahnost disappears. With 
added traffic ahead the display would be even more difficult to read. 

Pilot 3 : Some of the symbology occasionally becomes obscured and is difficult to read. 

Pilot 4 : Traffic, other than alerts and/or resolutions, clutter the ADI display. 

Pilot 5 : Surface traffic extremely hard to determine position until close to runway. 

TCAS traffic on ADI confusing transition to outside must be made at some point. 
Runway sensor vs data discrepancy hard to detect until close to ground. 

Pilot 6 : Disadvantages outweigh the advantages - too much of the wrong information 
(symbology, size) and the primary required information (symbology) is too small. 

Pilot 7 : Too much information in this display and of wrong relative size make it too hard 
to interpret. 

Pilot 8 : Runway symbology is too small and aircraft incursions are difficult to detect 
initially, especially on Runway 8L. 

20. What improvements would you suggest for this display system? 

Pilot 1 : no comments given. 

Pilot 2 : A much larger ADI so as to increase the runway environment scale. 

Pilot 3 : Changes to some of the symbology (i.e., the ground traffic symbol and runway 
symbol (sensor and inertial)). 
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Pilot 4 : Remove traffic symbols from the ADI unless the traffic is an alert or resolution. 
Keep all traffic display on the HSI (ND). 

Pilot 5 : Remove TCAS from ADI. Enlarge ADI to accept larger symbology. 

Pilot 6 : Increase the size of the fit. director and incorporate a loc/gs cross for that portion 
of the approach (PFD). Decrease size or remove from the PFD all airborne traffic 
info/symbology. Decrease or remove from the PFD all ground traffic symbology. 

Pilot 7 : If NAV display was a larger screen perhaps TCAS icons could be relative size. 
Surface airport traffic could be small diamond. 

Pilot 8 : Reduce TCAS symbology - leave the color the same. Increase runway and 
autopilot/flight director displays as much as possible. Delete turn and slip indicator. 

21. Was there any symbology in this display that either appeared confusing or should be 
changed? 

Pilot 1 : no. 

Pilot 2 : Due to the scale there was some confusion as to whether a runway incursion was 
taking place. 

Pilot 3 : no comments given. 

Pilot 4 : The symbology for traffic on the ground was difficult to get used to. I would 
like to see a motion trend arrow inside the symbol. I would also add raw data to the 
flight path vector symbol. 

Pilot 5 : Traffic on ground. TCAS on ADI not very useful nor is there any depth of field. 
Seem to get in way of f/d a/p guidance. Discrepancy in runway from sensor data hard to 
discern from a distance. 

Pilot 6 : None that was confusing. See 20 above for changes. 

Pilot 7 : Ground traffic should only be displayed if it is possible conflict to a landing or 
takeoff aircraft. 

Pilot 8 : Only runway incursion problems associated with runway 8F. 

22. Suggestions/comments: 

Pilot 1 : no comments given. 
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Pilot 2 : The scale of the presentation is simply too small. If the display conld be 
enlarged the presentation would be acceptable. 

Pilot 3 : Good idea to use the probe scenario. I think it could be very easy to get 
complacent doing the same runs. The test pilot needs to be awakened some times. 

Pilot 4 : no comments given. 

Pilot 5 : Not much different than current displays in use. 

Pilot 6 : The basic concept for this display is good. The amount and size and timing of 
displayed symbology need to be reworked. 

Pilot 7 : Items 17-21. 

Pilot 8 : This display concept could be easily incorporated into present Boeing 757/767 
EFIS equipment and would enhance the present system by improving TCAS data and 
runway incursion possibilities during low visibility approaches. 
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Display Evaluation 


(check one) 

Conventional HUD (CH) 

Enhanced HUD (EH) 

Embedded Conventional (EC) 

V Pictorial Vision (PV) 

1. The display yon have been monitoring is now to be evaluated over all of the 
experimental conditions encountered by using the modified Cooper-Harper rating 
scale shown below. In determining the Cooper-Harper operator rating for this 
display, it is important to stall at the "operator decisions" block located at the 
lower left of the chart. 



Cooper-Harper Rating Mean = 1.625 
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2. In the previous question, you determined the Cooper- Harper operator rating 
number for this display. When you selected the Cooper- Harper rating number 
among the final group of three (with the exception of a Cooper-Harper rating of 
10), what adjective description listed under "Difficulty Level/Operator Demand 
Level" influenced your decision? Please discuss. (Example: I selected a rating of 
3 because the symbology layout provided ...) 

Pilot 1 : Pictorial vision display provided excellent symbology, which seemed relatively 
easy. I became more comfortable with it after numerous approaches. 

Pilot 2 : One rating was selected because the display was very close to reality. Very little 
translation was required; accurate situational awareness. 

Pilot 3 : “1” very easy and useful display decreases workload tremendously. 

Pilot 4 : What appeal's to be complete situational awareness. The field of view is very 
good and traffic alerts and resolutions are noticed very quickly. It almost gives a three 
dimensional effect. The pathway presents a fulltime unconscious evaluation of LOC and 
G/S performance which reduces the amount of crosschecking required of the raw data. 
This reduction in workload really pays off when unexpected problems arise which 
demand additional attention. One can allocate that additional time while still 
subconsciously monitoring the approach. 

Pilot 5 : Selected 2 because display provided attainable performance with a minimum of 
mental effort. While this display was close to a one rating the mental effort still required 
was more than a minimal effort. 

Pilot 6 : With the exception of a few minor design changes discussed at the end, I felt the 
display was very user friendly - Flight guidance was simple, easy to interpret and follow. 
Overall SA was greatly improved over conventional systems. Easier to monitor not only 
flight progress, but the complete flight environment. I believe this alone lends itself to a 
safer operation for both the user a/c and airspace it’s operating in. 

Pilot 7 : Selected 1. Excellent display. Wide field of view provides less interpretation, 
and thus mental workload. 

Pilot 8 : I selected a rating of 2 because the symbology graphics were the best of any 
display. They are bright, clear and easy to read and react to . This display just needs to 
be reduced and brightened in the map screen as well as reduce the airspeed indicator and 
move it inside the left field screen boundary along with moving the glide path indicator 
inside the right field screen boundary. 

17. How would you assign crew role/duties in using this system? 
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Pilot 1 : P/F: fly the a/c, decision making (CAPT does the autoland at UNITED): PNF: 
comm/callouts/monitor autoland functions, etc. 

Pilot 2 : Pilot Flying - Monitor LOC and GS and runway environment. Pilot Not Flying 
- Monitor airspeed; call altitudes; monitor other traffic. 

Pilot 3 : PF 75% PV 25% outside. PNF 75% outside 25% PV. 

Pilot 4: PF stay inside until ~ 100 feet then begin to combine inside and outside scans. 
PNF stay inside for entire approach and landing. 

Pilot 5 : Would still assign duties as done on current systems with a pilot flying and a 
pilot not flying who is backing up and monitoring the flying pilot who is in turn 
monitoring the aircraft. 

Pilot 6 : PF - Monitors a/c performance as it pertains to the approach (i.e., alt, as, loc, gs, 
a/p and fit direc.) PNF - monitors any airborne or gnd traffic. Primary concern should 
be airport/runway environment while crosschecking outside for a visual on the landing 
runway. 

Pilot 7 : Similar to current roles, i.e. Pilot flying - flying aircraft, pilot not flying - 
monitors makes callouts - calls deviations. 

Pilot 8: Same as the HUD down without sensors in the VMC environment only. Crew 
duties always vary significantly based on VMC or IMC conditions. 

18 Please discuss the advantages of this display concept: 

Pilot 1 : Total ‘3d’ picture... very easy to get used to, felt more comfortable with it as we 
flew more approaches. 

Pilot 2 : This depicts a simulated VFR environment. Excellent situational awareness. 

Pilot 3 : Easy to read, nice to look at good and helpful information. 

Pilot 4 : It provides the big picture in one location. Reduces the amount of mental 
“picturing” of a situation. 

Pilot 5 : Easy to determine relative position to required flight path and any divergence 
from path. Easy to pickout TCAS traffic position. Failures are relatively easy to 
determine. Overall situational awareness is high with a minimum output of effort. It 
would be easy to monitor an autoland with this display in relation to today’s monitored 
autoland approaches. 

Pilot 6 : “A picture is worth 1000 words.” Fess mental work (i.e., no need to read and 
interpret numerous analog instruments to build a mental picture.) Overall situational 
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awareness of your entire flight environment is there in front of you. Eliminates guess 
work of where you are in the sky, where other traffic is in relationship to your a/c. Ease 
of use and monitoring allows for better multiple cockpit tasking. Provides increased 
safety because of the above listed items. 

Pilot 7 : This display provides “the big picture” allowing the pilot to consider all aspects 
of his current environment. 

Pilot 8 : The graphics displays are the best - very clear and bright, especially the TCAS 
information. I also like the ILS path display and the runway incursion examples. 

19. Please discuss the disadvantages of this display concept: 

Pilot 1 : none. 

Pilot 2 : The only minor problem was the location of airspeed a little difficult to include 
in the scan. 

Pilot 3 : No engine indications. 

Pilot 4 : None noticed. 

Pilot 5 : Sometimes difficult to tell direction of TCAS traffic while turning. Pitch 
oscillations vs autopilot/f/d errors sometimes require a “trend” before determining that an 
error has occurred. Possible hypnotic effect caused by tunnel. 

Pilot 6 : Potential exists to have a very cluttered display in a high density traffic area. 
Numerous airborne and ground traffic being displayed could disrupt (PF) and 
unnecessarily increase workloads at critical flight times (i.e., takeoff and landings) as 
well as cover or blank critical flight displays. 

Pilot 7 : no comments given. 

Pilot 8 : Although I like the ILS path display it tends to make the screen very cluttered on 
the turn from base to final approach when your trying to identify runway location and 
possible ground traffic. 

20. What improvements would you suggest for this display system? 

Pilot 1 : no comments given. 

Pilot 2 : Display airspeed in the area of the F/O and A/P. 

Pilot 3 : no comments given. 
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Pilot 4 : Once capturing the LOC and GS have the raw data lines (outside of the FD) turn 
green and flash, then go steady. Conversely, if there is a failure of LOC/GS, turn that 
lure red and flash it before removing the line. 

Pilot 5 : no comments given. 

Pilot 6 : Limits to when and how much information will have to be incorporated into the 
software so that condition of #19 will not affect the safe operation of the flight. 

Especially during critical phases of flight. 

Pilot 7 : The system could have a monitoring between the IRS runway symbol graphic, 
i.e. lat/long and magenta runway sensor position. When not in limits magenta runway 
symbol would flash - to better, faster interpretation. 

Pilot 8 : This is my favorite display however I would reduce and brighten the map 
screen. Also reduce and move the airspeed indicator inside the field screen and move the 
glide path on the right inside the altimeter. 

21. Was there any symbology in this display that either appeared confusing or should be 
changed? 

Pilot 1 : No. 

Pilot 2 : No. 

Pilot 3 : DME readout should be relocated to a corner in the display. 

Pilot 4 : The TCAS symbology needs to be dynamically sized in relation to distance. 
Currently the traffic on the opposite runway, which is no threat, displays a large symbol 
and detracts from the flight data. Also, other traffic, which is of no threat, attracts 
attention when displayed on screen for short periods. Non-threat traffic displays on the 
ND are great and might be enough data. 

Pilot 5 : Size of TCAS traffic display. F/d annunciator on top of display was a little 
difficult to bring into overall scan. Required a look up, outside of the normal scan to 
gather the info. 

Pilot 6 : Yes - the DME is hard to locate. Suggest it be boxed and placed below the a/s 
scale on the left side of the display (similar to RA on the right). Loc and gs needles 
should be fixed and not attached to the flight director. Alt and a/s scales should not float - 
tied to the heading and cmd guidance boxes at the bottom and top of the display. 

Pilot 7 : The magenta dashed approach corridor symbology could be larger dashed lines. 
This would give a better indication of possible loc or gs error. 
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Pilot 8 : Possibly change the color of the actual/real railway on the ILS display so they 
don’t both use the same color. The final ILS path and railway outline are very similar 
when you first line up on final. 

22. Suggestions/comments: 

Pilot 1 : no comments given. 

Pilot 2 : The best display I have ever seen. 

Pilot 3 : no comments given. 

Pilot 4 : The shading of the real world, as displayed through the ND sector, altimeter and 
VS; is about two notches too dark. I would like to see it slightly brighter. I would like to 
see a partial compass rose on the ND. I’d like to see the wind direction/speed just below 
the flight director. 

Pilot 5 : no comments given. 

Pilot 6 : A/p excursions could be difficult to catch early - should be coupled to an 
automatic a/p disconnect if a/p is unable to follow course guidance. For the scenario 
where the a/p and sensor data were in conflict - requires interpretation of more data 
and/or symbology to determine what and how large the problem is. This I believe can be 
overcome with training. 

Pilot 7 : Item 20. 

Pilot 8 : For all OLD pilots like myself I support an mandatory 30 minute nap after lunch 
before any flying activity. 
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PROBE QUESTIONNAIRE 


(check one) 

/ Conventional HUD (CH) 

Enhanced HUD (EH) 

Embedded Conventional (EC) 

Pictorial Vision (PV) 

The experimenter will complete the probe questionnaire based upon the verbal comments obtained from the 
evaluation pilot. This shall be done immediately after the blanking of the screen. 

Question: 

1 . At the moment the screen went blank, what was your overall situation? 

Pilot 4 : TCAS advisory with a/c on parallel runway; GS flag on EADI. 

Didn’t notice removal of glideslope bar. 

Pilot 6 : Lost glideslope. Immediately disconnected AL approach. 

3. At the moment the screen went blank, you were: 

right of path P6 

left of path 

on path P4 


4. At the moment the screen went blank, you were: 

above the glideslope 

below the glideslope 

on the glideslope P4: P6 


6 . 


CONVENTIONAL AND PICTORIAL 

a. Your lateral error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2-2 dot error units 

P4 P6 

b. Your vertical error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

P4 

P6 
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8 . 


Mention any other anomalies yon may have detected. 


Pilot 4 : none. 

Pilot 6 : Looked like initially AP not following FD. Then loss of GS. 

9. If this had been an actual flight situation, what would you have done? (prioritize) 

Pilot 4 : Disconnect a/p since VMC. Given more time, evaluate traffic alert 
and adjust situation. Take care of traffic conflict first. 

Pilot 6 : Disconnect a/p and manually fly the visual landing. 

10. Evaluate this display for ease of maintaining "spatial awareness" while monitoring the approach. 

very somewhat neutral somewhat very 

hai’d hai’d easy easy 

P4 P6 

11. Suggestions and/or comments concerning the probe scenario. 

Pilot 4: When getting localizer or g/s flag failure would like to see flashing 
indication (even though in today’s a/c this flashing doesn’t exist. 

Pilot 6: Very useful since subject has not been cued into looking and 
memorizing status just in case this scenario comes up. “Perfectly legal”. 
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PROBE QUESTIONNAIRE 


(check one) 

Conventional HUD (CH) 

✓ Enhanced HUD (EH) 

Embedded Conventional (EC) 

Pictorial Vision (PV) 

The experimenter will complete the probe questionnaire based upon the verbal comments obtained from the 
evaluation pilot. This shall be done immediately after the blanking of the screen. 

Question: 

1 . At the moment the screen went blank, what was your overall situation? 

Pilot 7 : Initially F/D command continued turn to left and a/p wasn’t following. Saw 
g/s flags and so TOGA’d. 

Pilot 8 : Left hand descending, left bank (8°), 5° pitch up. Intercepting final 
approach with G/S, LOC captured. (Was looking at heads down display when 
screen blanked). 

3. At the moment the screen went blank, you were: 

right of path P7 P8 

left of path 

on path 

4. At the moment the screen went blank, you were: 

above the glide slope P7 

below the glideslope 

on the glideslope P8 

6. CONVENTIONAL AND PICTORIAL 

a. Your lateral error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

P8 P7 

b. Your vertical error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

P7 

P8 
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8 . 


Mention any other anomalies yon may have detected. 


Pilot 7 : none. 

Pilot 8 : Lost G/S. 

9. If this had been an actual flight situation, what would you have done? (prioritize) 

Pilot 7 : Would have TOGA’d and continued left hand turn to stay away 
from parallel track.. 

Pilot 8 : Called ATC for “missed approach” (lost g/s in IMC), turn toward 
runway and climb to MA altitude on runway heading. 

10. Evaluate this display for ease of maintaining "spatial awareness" while monitoring the approach. 

very somewhat neutral somewhat very 

hai’d hai’d easy easy 


P7 

P8 

11. Suggestions and/or comments concerning the probe scenario. 

Pilot 7 : Has learned points at which scenarios occur and was fixating on a/p 
and f/d. Loss glideslope and localizer error indicators because of fixations. 
Typically go to HUD after rolled out on final. No comments/suggestions 
for the probe. 

Pilot 8: Just feels that this scenario shows why one crew member must 
monitor a/c instruments at all tunes. Good scenario for finding out what 
pilots will do when you lose main instruments and have to use stand-by 
instruments. 
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PROBE QUESTIONNAIRE 


(check one) 


Conventional HUD (CH) 

Enhanced HUD (EH) 

/ Embedded Conventional (EC) 

Pictorial Vision (PV) 

The experimenter will complete the probe questionnaire based upon the verbal comments obtained from the 
evaluation pilot. This shall be done immediately after the blanking of the screen. 

Question: 

1 . At the moment the screen went blank, what was your overall situation? 

Pilot 1: TCAS advisory, lost G/S, before that having tough time with circles moving 
apart (that was the 1st concern). 

Pilot 3: AP not following FD, TA, Loss of LOC. 


3. At the moment the screen went blank, you were: 

right of path P3 

left of path 

on path PI 

4. At the moment the screen went blank, you were: 

above the glide slope 

below the glideslope P3 
on the glideslope PI 


6. CONVENTIONAL AND PICTORIAL 

a. Your lateral error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

PI P3 

b. Your vertical error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

PI P3 
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8 . 


Mention any other anomalies yon may have detected. 


Pilot 1 : Just circles chasing each other although needles were pretty much 
on. 

Pilot 3 : no comments given 

9. If this had been an actual flight situation, what would you have done? (prioritize) 

Pilot 1 : TCAS was a major concern (that close in). Almost ready to 
“wave it off’ (go around). At loss of g/s would have already done it. 

Pilot 3 : Taken over manually and broken off approach (GA) and gone left 
(to avoid traffic that generated TA). Ask for clearance for special 
approach (NON localizer). 

10. Evaluate this display for ease of maintaining "spatial awareness" while monitoring the approach. 

very somewhat neutral somewhat very 

hai'd hai'd easy easy 


PI P3 

11. Suggestions and/or comments concerning the probe scenario. 

Pilot 1 : Good scenario. Some of those things happen in real life (traffic landing 
on wrong runway. 

Pilot 3 : Good scenario, if at least to break monotony. 
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PROBE QUESTIONNAIRE 


(check one) 


Conventional HUD (CH) 

Enhanced HUD (EH) 

Embedded Conventional (EC) 

V Pictorial Vision (PV) 

The experimenter will complete the probe questionnaire based upon the verbal comments obtained from the 
evaluation pilot. This shall be done immediately after the blanking of the screen. 

Question: 

1 . At the moment the screen went blank, what was your overall situation? 

Pilot 2 : TA right in front - pressed Concern, lost GS but wan on GS when it went 
out. 

Pilot 5 : TCAS advisory and had g/s failure. Had no vertical guidance and that’s 
what he’s concerned about. Concern press for TCAS and TOGA for g/s failure. 

3. At the moment the screen went blank, you were: 

right of path P5 

left of path 

on path P2 

4. At the moment the screen went blank, you were: 

above the glide slope 

below the glideslope 

on the glideslope P2: P5 


6. CONVENTIONAL AND PICTORIAL 

a. Your lateral error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

P2 P5 

b. Your vertical error was : 0 - 1/4 - 1/2 - 3/4 - 1 - 1 1/2 - 2 dot error units 

P2 P5(l/8) 
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8 . 


Mention any other anomalies you may have detected. 


Pilot 2 : None. 

Pilot 5 : None. 

9. If this had been an actual flight situation, what would you have done? (prioritize) 

Pilot 2 : TOGA because of GS and executed G/A. G/S should have cleared 
TA traffic since he was below us. Advise tower. 

Pilot 5 : Disconnect a/p and make a climbing right turn. 

10. Evaluate this display for ease of maintaining "spatial awareness" while monitoring the approach. 

very somewhat neutral somewhat very 

hai’d hai’d easy easy 

P5 P2 

11. Suggestions and/or comments concerning the probe scenario. 

Pilot 2 : Nervous when answering questions. 

Pilot 5 : To make it a little harder, throw in some talking. In actual flight 
situation, typically have person talking, something happening and you’re 
flying. Impression is that other a/c is coming into our path. 
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Final Questionnaire 


(note: The average rating (of the 8 evaluation pilots) for each display concept is shown 
above the rating scale.) 

1. Based upon exposure to the three cockpit displays flown during this 
experiment in IMC conditions, please indicate your OVERALL relative 
ranking/grading of the displays by placing "EH", "EC”, or "PV" at the 
appropriate location on the horizontal line shown below. In determining 
the overall evaluation, consider all monitoring tasks, i.e., the approach, 
verifying the location of the runway, detecting ground runway incursions, 
etc. 

where: 

"EH": indicates Enhanced HUD (IMC) 

"EC": indicates Embedded Conventional (HUD info in PD) 

"PV": indicates Pictorial Vision 


EXAMPLE 



EH 


1 1 1 


1 

PV 

1 1 

EC 

1 1 

1 1 1 

1 2 

3 

4 

5 6 

7 8 

9 10 

least desirable 





most desirable 

display 
1 1 1 


1 

EC(5.75) 
1 1 

EH(7.75) 
1 1 

display 

PV(9.38) 

1 1 1 

1 2 

3 

4 

5 6 

7 8 

9 10 

least desirable 





most desirable 

display 





display 


2. Discuss your rationale in determining the overall evaluation in item 1 above. 

Pilot 1 : CH superb piece of equipment that allows pilot to remain out of cockpit, looking 
for traffic, etc... EH also exceptional with a lot of info that is very helpful. 

Pilot 2 : EC was such a small display that some information was covered, e.g. a/c on 
taxiway was blocked by f/o a/p display. EH was an excellent display but did not display 
other aircraft in TCAS adequately. PV had virtually no flaws. 

Pilot 3 : Taking into consideration overall ease of use and information available. 
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Pilot 4 : PV has a much wider field of view with better situational awareness. EC lacked 
FOV but made up for some of that by centrally locating more info in one small area. EH 
required additional crosschecking over large distances and additional time to comprehend 
symbolics due to use of monochrome. 

Pilot 5 : PV was easy to monitor and all info was readily available. Conditions were 
closest to actually looking out of the window. Guidance was easy to determine due to 
tunnel so interpretation of pathway deviations was almost instantaneous. Felt TCAS 
symbology could be worked on, to present a more realistic aircraft icon with better depth 
of field and less obtrusive on display. Enhanced HUD was fairly easy to use except for 
the continuing transition from top to bottom display. Can sometimes miss information 
on one display while looking at the other. If all info is repeated on both displays then 
could get rid of one display stopping the transition (of course then you would probably 
have a PV). EC display was a somewhat dated display that didn’t provide much more 
than today’s displays. Easy to assimilate all it’s info but a little confusing to interpret 
ADI info such as TCAS, runway, traffic info due to size of display. 

Pilot 6 : I’m a picture book kind of guy! It’s easier for me to see a picture, than have my 
weak brain try to create a picture derived from gauges and analog information. I strongly 
believe a HUD will enhance any system or display of flight guidance and could even be 
incorporated with the PV display. The embedded conventional HUD in my 
determination would only be useful in an emergency situation (i.e. loss of HUD) - the 
information could then be displayed in a redundant format on the PFD - but not as a 
primary mean’s for displaying traffic and navigational approach information. 

Pilot 7 : The embedded conventional was not a system that would make the pilot 
workload less. It required too much mental interpretation. Moving up this scale EH was 
an improvement with Pictorial Vision at the top improving on the other two systems 
almost to perfection. When comparing all three systems PV was only system that would 
make a CAT IIIc a comfortable approach. 

Pilot 8 : The EH simply incorporates the sensors and the HUD making it the best display, 
however PV graphics are the best and could improve EH especially in the TCAS info, 
area. EC is equally good however runway and incursion graphics need to be expanded. 
PV data needs to be reduced and the map screen could be brighter. 


Rank/Grade the three displays flown in IMC conditions with respect to the following 
factors. 


where: 

"EH": indicates Enhanced HUD (IMC) 

"EC": indicates Embedded Conventional (HUD info in PD) 
"PV": indicates Pictorial Vision 
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3. 


improving flying safety 


EC(6.38) EH(7.75) PV(9.13) 


1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

monitoring an "autoland" in restricted visibility conditions 

EC(6.63) EH(8.13) PV(9.50) 

1 1 1 1 1 I 1 1 1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

monitoring other traffic 

EH(6.75) 

EC(6.88) 

1 1 1 1 1 1 1 

1 

PV(8.88) 

1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

allowing you to think ahead of the aircraft 

EC(5.75) 

1 1 1 1 1 1 1 

EH(8.13) PV(9.63) 

1 1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

presenting sensor-based information 

EC(6.63) 

1 1 1 1 1 1 1 

EH(8.13) PV(9.50) 

1 1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 

recognizing aircraft runway incursions. 

EC(4.63) EH(7.88) 

1 1 1 1 1 1 1 1 

PV(9.50) 

1 1 1 

1 2 3 4 5 6 7 

least desirable 
display 

8 

9 10 

most desirable 
display 
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9. detecting a Flight Director and F/D Raw Data conflict. 

EC(6.88) EH(7.63) PV(9.13) 


1 2 3 

4 

5 6 7 8 9 10 

least desirable 


most desirable 

display 


display 

10. understanding a Flight Director and F/D Raw Data conflict. 

1 1 1 

1 

EC(7.00) EH(8.25) PV(9.25) 

1 1 1 1 1 1 1 

1 2 3 

4 

5 6 7 8 9 10 

least desirable 


most desirable 

display 


display 

1 1 . detecting the scenario where the autopilot stops following the F/D 

1 1 1 

1 

EC(6.88) EH(8.25) PV(9.00) 

1 1 1 1 1 1 1 

1 2 3 

4 

5 6 7 8 9 10 

least desirable 


most desirable 

display 


display 

12. understanding the scenario where the autopilot stops following the F/D 

1 1 1 

1 

EC(7.13) EH(8.00) PV(9.13) 

1 1 1 1 1 1 1 

1 2 3 

4 

5 6 7 8 9 10 

least desirable 


most desirable 

display 


display 

13. detecting an autopilot (assume a navigational system error) and sensor 

conflict. 

1 1 1 

1 

EC(6.63) EH(7.75) PV(9.38) 

1 I 1 1 1 1 1 

1 2 3 

4 

5 6 7 8 9 10 

least desirable 


most desirable 

display 


display 

14. understanding 

an 

autopilot (assume a navigational system error) and 

sensor conflict. 
1 1 1 

1 

EC(6.00) EH(7.63) PV(9.38) 

1 1 1 1 1 1 1 

1 2 3 

4 

5 6 7 8 9 10 

least desirable 


most desirable 

display 


display 
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15. detecting an autopilot oscillation. 


EC(7.63) EH(8.63) 

PV(9.25) 


1 2 3 

4 

5 

6 7 

8 

9 10 

least desirable 





most desirable 

display 





display 


16. understanding an autopilot oscillation. 

EC(7.75) 

EH(8.63) 

PV(9.25) 


1 2 3 4 5 6 

least desirable 
display 

7 

8 

9 10 

most desirable 
display 

17. making the decision to go-around 

EC(6.13) EH(7.75) 

1 1 1 1 1 1 1 1 

PV(9.38) 

1 1 1 

1 2 3 4 5 6 

least desirable 
display 

7 

8 

9 10 

most desirable 
display 

18. reducing overall workload 

EC(5.88) 

1 1 1 1 1 1 

1 

EH(8.25) PV(9.63) 

1 1 1 1 

1 2 3 4 5 6 

least desirable 
display 

7 

8 

9 10 

most desirable 
display 

19. maintaining situation awareness* 

EC(6.25) 

1 1 1 1 1 1 1 

EH(8.25) PV(9.75) 

1 1 1 1 

1 2 3 4 5 6 

least desirable 
display 

7 

8 

9 10 

most desirable 
display 


Note: One could define situation awareness as "...the pilot has an 

integrated understanding of the factors that will contribute to the safe 
flying of the aircraft under normal or non-normal conditions." 
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(note: For questions 20- 27 , the scale ranges from 1 to 5 with strongly disagree =1 and 
strongly agree =5 ) 

20. The Enhanced HUD (EH) system is a useful concept. 


strongly disagree 

neutral 

agree strongly 

disagree 

□ 

agree 


average pilot rating is 4.38 


21. The Embedded Conventional (EC) system is a useful concept. 

strongly disagree neutral agree strongly 
disagree agree 


average pilot rating is 3.50 

22. The Pictorial Vision (PV) system is a useful 

strongly disagree neutral agree 

disagree 

□ □ □ □ 

average pilot rating is 5.00 

23. I would feel comfortable flying with the sensor information 

presented in the Enhanced HUD (EH) concept. 

strongly disagree neutral agree strongly 

disagree agree 


concept. 

strongly 

agree 


average pilot rating is 4.38 
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24. I would feel comfortable flying with the sensor information 
presented in the Embedded Conventional (EC) concept, 
strongly disagree neutral agree strongly 
disagree agree 


average pilot rating is 3.25 

25. I would feel comfortable flying with the sensor information 
presented in the Pictorial Vision (PV) concept. 

strongly disagree neutral agree strongly 

disagree agree 

□ □ □ □ □ 


average pilot rating is 5.00 

26. The sensor information is needed for landing in restricted visibility 
conditions. 


strongly disagree neutral agree strongly 
disagree agree 


average pilot rating is 4.63 


27. The sensor information could increase flight safety. 


strongly disagree 
disagree 


neutral 



agree 


strongly 

agree 


average pilot rating is 4.75 

28. Comments/suggestions concerning this experiment: 

Pilot 1 : Interesting to fly with the different cockpit displays! Both the conventional HUD 
and enhanced HUD would greatly aid the pilot regarding flight safety and situational 
awareness. Enjoyed working with everyone - hoped I helped! 

Pilot 2: The program is well organized and presented. At the end of day one I was quite 
fatigued and made mistakes I might have caught earlier. Either a shorter 1 st day or a 
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change of activity at the end of the day might help performance. 

Pilot 3 : Re: The monitoring on PV. Did not see traffic until within field of view. 
Although traffic would be displayed on nav display in lower left corner. It seemed as 
though I would not refer to that display very often. Traffic was easier to monitor on 
conventional nav display. 

Pilot 4 : None except for those comments already made on the individual questionnaires. 

Pilot 5 : During times when the eight ball ( the backup instrument represented by the ADI 
portion of the EFIS display format) appeal's return the display to the original situation 
and allow pilot to return aircraft to path in an effort to determine how easily a display is 
to interpret gross deviations followed by a correction. 

Pilot 6 : no comments written down. 

Pilot 7 : Great set up. It might be better if each approach display was different, if this is 
an option. Overall a well run/important experiment. Take it from someone who would 
like to fell better/have a higher comfort level on a CAT III approach! 

Pilot 8 : The sensor information is a must for any future aircraft instrumentation for 
increased flight safety first and restricted visibility conditions approaches and landings. 
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